6.11.1: Some dependencies are no longer being installed

What Happened

It appears that running npm install with NPM 6.11.1 isn’t downloading some dependencies that were downloaded with NPM 6.10.3.

Reproduction Steps

git clone https://github.com/WordPress/gutenberg.git gutenberg
cd gutenberg
npm install
npm run check-local-changes

Repeating these instructions with NPM 6.10.3 works as expected. (After reverting changes to package-lock.json, and removing node_modules.)

Platform Info

$ npm --versions
{ gutenberg: '6.3.0',
  npm: '6.11.1',
  ares: '1.15.0',
  brotli: '1.0.7',
  cldr: '35.1',
  http_parser: '2.8.0',
  icu: '64.2',
  modules: '64',
  napi: '4',
  nghttp2: '1.39.2',
  node: '10.16.3',
  openssl: '1.1.1c',
  tz: '2019a',
  unicode: '12.1',
  uv: '1.28.0',
  v8: '6.8.275.32-node.54',
  zlib: '1.2.11' }
$ node -p process.platform
darwin

Reproduced. Also on Mac.

(Large install so not easy to tell what is going on. I noticed npm ls shows missing peer dependencies even with npm v6.10.3.)

I compared the log of 6.11.1 and 6.10.3 and the first package difference was 6.11.1 log was missing plugin-transform-reserved-words.

npm info lifecycle @babel/plugin-transform-reserved-words@7.2.0~preinstall: @babel/plugin-transform-reserved-words@7.2.0
# 6.10.3
$ npm ls '@babel/plugin-transform-reserved-words'
gutenberg@6.3.0 /Users/john/Documents/Sandpits/npm.community/9586/gutenberg
└─┬ @wordpress/babel-preset-default@4.4.0 -> /Users/john/Documents/Sandpits/npm.community/9586/gutenberg/packages/babel-preset-default
  └─┬ @babel/preset-env@7.4.4
    └── @babel/plugin-transform-reserved-words@7.2.0 

# 6.11.1
$ npm ls '@babel/plugin-transform-reserved-words'
gutenberg@6.3.0 /Users/john/Documents/Sandpits/npm.community/9586/gutenberg
└── (empty)
1 Like

Hi! I just want to post another confirmation of this change in behavior. For what it’s worth, I can clarify that I think this is a regression, not a desirable change.

Shields ships with a local dependency:

    "gh-badges": "file:gh-badges",

Typically npm ci and npm install have installed the dependencies declared in gh-badges/package.json such as is-css-color. Recently this stopped working on Heroku, which I’m guessing happened when npm 6.11 was released. It can be reproduced locally by checking out the Shields repo and running:

npm install -g npm@6.11.2
rm -rf node_modules
npm install
ls -d node_modules/is-css-color

After a lot of experimentation it looks like npm install -g npm@6.10.3 does not have this problem.

We typically use npm ci which is fine in either version, but for whatever reason, that is not the command that the Heroku Node buildpack uses, and same goes for many devs who pop into the project.

We’re pinning npm to 6.10.3 for now: https://github.com/badges/shields/pull/3904

1 Like

This is a result of the 2 commits that fixed reinstall breaks after npm update to 6.10.2 and Installing the same module under multiple relative paths fails on Linux

A more minimal reproduction case: https://github.com/isaacs/npm-test-file-metadeps/ (Clone and run npm test.)

The behavior change is that dependencies of linked file: dependencies are not listed in the package-lock.json file. So, they’re not installed in node_modules on subsequent installs.

This is actually a weird bit of behavior if you think it through.

A file: dep may be anywhere. In these cases, it’s local in the project, so it’ll be able to load its dependencies from node_modules. However, if it was file:../gh-badges, then it wouldn’t. And, if it had dependencies locally in its tree, then those dependencies would not find their deps further up, either.

Logically, a linked dep needs to be treated as a completely independent top-of-tree, and will be in npm v7. (I wrote about this recently on the npm blog.) I was thinking there about having a --deep option to npm install, to tell it to also install the children of linked deps (in their own node_modules folder). (Or maybe make it enabled by default, and add a --shallow to say “don’t bother”.)

I’m going to investigate if it’s possible to list child deps in a way that does not regress the problem that it was intended to fix.

It’s a bit surprising to me that it’s installing the metadeps in absence of a package-lock.json, but only not saving them to the package-lock.json. So at the very least, that is a bug. It should be saving what it does reliably so that it can be repeated.

My apologies, this section of the code is fairly brittle, which is why I intend to replace it.

4 Likes

Thanks for the investigation, @isaacs!

Gutenberg has grown fairly complex in its dependency tree, so I’m not entirely surprised things have broken in weird and unexpected ways.

Good to hear that this is going to be tightened up in npm v7! If you have any suggestions for making our config less dependent on surprising edge behaviour, I’m happy for us to implement them in the mean time.

I’ve just spent some time testing this and the regression in npm was indeed introduced in 6.11.1

Specifically reverting the following change resolves this issue:

Via npm/cli/compare/npm:53cf5e9…npm:6c18f1f
~/.nvm/versions/node/v10.16.3/lib/node_modules/npm/node_modules/cmd-shim/index.js

Removing the : reference from both instances of :find_dp0 fixes the issue.

Running the following repro steps results in no changes to the package-lock.json file:

git clone https://github.com/WordPress/gutenberg.git gutenberg
cd gutenberg
npm install
npm run check-local-changes
1 Like

Thanks so much for the detailed information and the fix! 6.11.3 seems to be working well for us :+1:

I’ll be curious to see how this configuration will be supported going forward.

In Shields we’re shipping a dependency inside the repo for the application. The application has a single dev/prod lockfile that includes the subdependencies, and it’s automatically maintained by Dependabot. The right deps are also installed with npm ci and npm install. We recommend npm ci, though Heroku uses npm install, and many contributors may run that, even though we suggest npm ci.

It’s important that the lockfile include the gh-badges subdependencies as we want them pinned with the application.

I like it that both npm ci and npm install behave the same here in that they install the subdependencies by default. If --deep and --shallow are options and --deep is not the default, it would be nice if this were configured via package.json so we don’t have to try to train new contributors to npm install --deep.

If there’s anything else I can do to clarify (or to discuss a future solution) let me know!

6.11.3 is working for us, too. Thanks for the fast fix, @isaacs! :sparkling_heart: