`npm dedupe` after `npm install --global-style` dedupes incorrectly

What I Wanted to Do

I have a package whose dependencies were previously installed with --global-style. Among these dependencies, some reference different version of a specific dependency. I want to dedupe my package and expect everything to resolve correctly.

What Happened Instead

Some sub-dependencies are deduped and eventually resolved incorrectly. An example minimal repo will be provided.

Reproduction Steps

  1. Clone https://github.com/IanSavchenko/npm-dedupe-issue - it has a minimal package.json with two git dependencies.
{
  "dependencies": {
    "aws-util-firecloud": "git://github.com/tobiipro/aws-util-firecloud.git#semver:~1.5.5",
    "lodash-firecloud": "git://github.com/tobiipro/lodash-firecloud.git#semver:~0.2.0"
  }
}
  1. Run
npm install

(it will use global style, as set in .npmrc)

npm list --depth=0

should show something like

dedupe_test@1.0.0 /Users/isao/code/test/dedupe_test
├── aws-util-firecloud@1.5.6 (git://github.com/tobiipro/aws-util-firecloud.git#64585a405f3ec2217e3d3ef13c0395e8f236e05c)
└── lodash-firecloud@0.2.15 (git://github.com/tobiipro/lodash-firecloud.git#a62de48dfa114035e700474d6cd82290c667b1f4)
  1. Run npm dedupe - dependency tree gets flattened

  2. Check node_modules/lodash_firecloud/package.json - it has version 0.2.15, as per root package.json (~0.2.0) - CORRECT

  3. Check node_modules/minlog/package.json - it has a dependency on lodash-firecloud to be ~0.3.0, but instead does not have node_modules dir included, so will use the one from root node_modules - WRONG.

Note: minlog is a dependency of aws-util-firecloud

  1. Check node_modules/aws-util-firecloud/package.json - it has a dependency on lodash-firecloud to be ~0.3.1. Then it gets correctly resolved from node_modules/aws-util-firecloud/node_modules/lodash-firecloud and has version 0.3.15 - CORRECT

Details

One thing that can be noted: running npm install without global style ends up installing packages correctly.

Platform Info

$ npm --versions
{
  dedupe_test: '1.0.0',
  npm: '6.9.0',
  ares: '1.15.0',
  brotli: '1.0.7',
  cldr: '35.1',
  http_parser: '2.8.0',
  icu: '64.2',
  llhttp: '1.1.1',
  modules: '72',
  napi: '4',
  nghttp2: '1.38.0',
  node: '12.1.0',
  openssl: '1.1.1b',
  tz: '2019a',
  unicode: '12.1',
  uv: '1.28.0',
  v8: '7.4.288.21-node.16',
  zlib: '1.2.11'
}
$ node -p process.platform
darwin

I have similar problems when running NPM with --global-style. In my case it was really helping to run npm dedupe, after a npm install --global-style

Might be that dedupe is not run automatically by NPM, when --global-stye is specified on the installation? (given that they have exactly opposite purposes; the first to flatten and the second to nest the dependency tree)

Did you manage to find a solution/explanation?

Hi Igor,

Yes, npm install --global-style and npm dedupe do totally different thing. It seems like that behaviour of npm install (without global style) is closer to npm dedupe, but not completely the same, because npm install resolves everything correctly.

As a temporary fix, we ended up running npm install right after npm dedupe.

So you do something like npm install && npm dedupe && npm install ? :frowning:

Won’t running install after dedupe reset the changes made by the dedupe?
Or being a node_modules/ already present there, influences the install and provokes different results?
(that would be, IMHO, pretty creepy in its total unpredictable behaviour)


Just to mention something related, it might be useful:
I had problems in running npm install --global-style && npm dedupe with all the NPM-5 versions that I tried (edit: errata corrige; just till 5.6.0. From 5.7.1 a lot of bugs have been solved).
Dedupe with NPM5 (<5.7.0) would completely loose some dependencies (that were instead there just after the simple install); so there is some bug there.
Using a NPM4 or NPM6 solved that broken behaviour (or a NPM5 > 5.7.0).


Won’t running install after dedupe reset the changes made by the dedupe ?

It will not reset them, but will just fix missing packages. The resulting node_modules structure does not look to be the most efficient and small possible deduped tree of packages, but at least it works.


I will give here a bit more insight into how we get to what we get.

We have monorepo-like setup for our AWS Lambdas, where there is one common node_modules directory fetched in global-style and a set of lambdas, each having it’s own package.json + node_modules.

/root
|--.npmrc    <-- this one says to use global-style when installing deps
|--package.json     <-- here we have regular external links to npm git repos
|--node_modules/     <-- here we have global-style packages
|--lambdas/
   |--lambda1/
      |--package.json  <-- references packages from `root/node_modules` as `file..///`
      |--node_modules/  <-- has only symlinks to `root/node_modules`
      |--src/
      |--dist/
         |--node_modules  <-- will have `lambda1/node_modules` unsymlinked and deduped
   |--lambda2/
      |--package.json  <-- references packages from `root/node_modules` as `file..///`
      |--node_modules/

Lambda’s package.json have file://.. dependencies to the root node_modules, so we have common versions of packages for all Lambdas (they are just referencing root) and use less disk space while in dev. Packages in the root node_modules are installed with --global-style, so lambdas can just cherry-pick those packages it needs. This setup allows us to run lambdas locally, run tests and all the tooling. But when we want to deploy lambdas to AWS, we need to package them. That’s when we resolve file://... symlinks (basically, copy those global-style dependencies from root node_modules to lambda’s dist/node_modules) and try to run npm dedupe over it, so the packages structure should get flat and smaller.

Where do you run the npm install after npm dedupe? In the unsymlinked dist/ folder?
Not related to your explanation, but still wondering about this weird npm install changing result depending on the eventually-already-present content on node_modules/. As I said, it feels creepy that I have to run the install more than once in order to achieve the correct (?) result. :thinking:

However, yours is an interesting approach.

I am wondering how it works when the Lambda projects have really different dependencies but with still deps that are shared/common among all.
Doesn’t it happen to need different versions of the same library? Can you afford to update versions because one project needs it and be confident that it won’t break any of the other projects?

Apart for these doubts randomly popped out in mind just now, I think it is a nice implementation and I can see the value of such setup. :slightly_smiling_face:

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.