NPM Needs a more robust story for local multi-module development

(Josh Mc Donald) #1

NPM Needs a better solution for local development

Given the following layout of multiple modules in development:

  • /my-repo
    • /my-application
      • package.json
    • /my-lib
      • package.json

And given my-application depends on my-lib:

If I’m making changes to both of these projects, or I wish to make changes to my-lib and see the effect they have within my-application, I currently have two supported options:

  1. Publishing tagged versions to the global repository, or
  2. Using npm link

The first option works, but it is slow, requires constantly changing the tag name, consumes valuable community resources, and pollutes the global namespace if you’re not using private packages.

The second option (using a symlink) while it apparently works for most people most of the time, can produce significantly different results than what you would get by installing a “real” published dependency via the npm public repositories:

  1. The fileset is different, as all files located within the source tree are visible, not just those specified by files:, .gitignore, and .npmignore
  2. More importantly, due to the node module resolution algorithm, the dependencies of my-lib will be wrong:
    1. Concrete dependencies will be immune from npm’s resolution of version ranges and de-duplication
    2. Any devDependencies and locally-installed peer-dependencies will be present and resolved within the scope of my-lib instead of my-app.

The second problem can cause pretty serious difficulties, when it leads to duplicates of libraries that should not have multiple instances, such as react, and it also breaks TypeScript: Any @types/foo dev-dependencies become effectively duplicated with different identities, causing a bunch of nonsensical Expected a FooBarBaz, but found a FooBarBaz errors.

I’m not sure what the best solution to this problem is, but I have a proposal that might be acceptable:

  1. Add a flag to npm publish which performs all the steps of a publish, but then copies the artefact to a known location within the local filesystem’s npm cache
  2. Add the ability to (within the scope of my-application) temporarily set the resolution of an installed dependency to use the local artefact as the source instead of fetching it from the cloud or using a direct symlink as currently used by npm link.

The idea to use a known location instead of relative paths to the artefact produced by npm pack is so that it works for transitive dependencies also, in case your project is more complex than this simple example, perhaps you have several libraries and an application which all depend on a “common” or “base” library, and they might not all have the same relative paths to it.

(Rebecca Turner) #2

Our roadmap for the next 3-6 months has Workspaces (as seen in yarn), which I believe will address your needs here quite precisely.

(Josh Mc Donald) #3

This just seems like more symlinks, which are the cause of the problem.

(Rebecca Turner) #4

I would argue that it’s not symlinks that are the problem, but how they’re being used.

For example:

You’ll find that Workspaces deduplicate dependencies exactly like an ordinary install.

In fact, the only meaningful difference between Workspace installs and ordinary installs, I think, is the filtering of the files included in them. If you have a scenario where this causes problems in practice, I’d be very interested in learning about it?

(Josh Mc Donald) #5

The issues that have been giving me pain recently are caused by the fact that the devDependencies of “my-library” will be visible from its node_modules when bundling the app, instead of coming from the app’s node_modules (if they’re shared), which breaks TSC when they’re typedefs.

(Rebecca Turner) #6

When we have our RFC up for our workspaces plan I’ll link here as we’d very much like to if you foresee any issues with it. I think it’ll work past your concerns. (While there is some symlinking planned, there’s probably less than you imagine.)

(Josh Mc Donald) #7

Great, thanks. The main problem I encounter is the dev-deps being exposed via the dependants’ node_modules being used during resolution. This might be something that can be worked around in TS3+, due to more flexibility being available in .tsconfig for resolving modules that might make it easier to just skip any linking/install for dependencies you’re developing locally. I should get a chance to test this in a week or two, I’m hoping.

(Ben Brown) #8

I’m curious to hear an update on the status of workspaces support. Is that going to land before year end, do you think @iarna?

Something I’m really looking forward to. :slight_smile:

(Wmoulin) #9

Hi, it’s possible to have a reply for the status of workspaces support ?

(Jan Nicklas) #10

Lerna does a great job here - especially because it allows a very flexible mono repository structure using globs.

Please take a look at lernas feature maybe some of them can be merged into npm workspaces