package-lock.json keeps changing between platforms and runs

cli
priority:medium
triaged

(Thomas Grainger) #1

Hi, this is a cross-post from https://github.com/npm/npm 1 since the issue in question does not seem to have made its way here yet.

The issue is: https://github.com/npm/npm/issues/17722 14

To summarize, the way package-locking handles optional dependencies results in inconsistencies in the lock-file across different platforms, which adds unexpected churn and results in some dependencies not being locked if the lock-file was generated on the “wrong” platform.

Sorry for not following the reporting template, but everything is kind of already said in the original issue.

What I Wanted to Do

npm install with a stable package-lock.json

What Happened Instead

package-lock.json is different between platforms and sometimes between runs

Reproduction Steps

loads of examples in that github bug

Details

Loads of examples in that github bug

Platform Info

$ npm --versions
{ npm: '6.2.0',
  ares: '1.14.0',
  cldr: '33.1',
  http_parser: '2.8.0',
  icu: '62.1',
  modules: '64',
  napi: '3',
  nghttp2: '1.32.0',
  node: '10.8.0',
  openssl: '1.1.0h',
  tz: '2018e',
  unicode: '11.0',
  uv: '1.22.0',
  v8: '6.7.288.49-node.19',
  zlib: '1.2.11' }

$ node -p process.platform
linux

package-lock.json and optional packages
package-lock.json changes from one `npm install` to the next
(Thomas Grainger) #2

Reformatted from package-lock.json and optional packages


(Thomas Grainger) #3

I think this is also related: https://github.com/npm/npm/issues/18103

I’m opening this issue because:

  • npm is crashing.
  • npm is producing an incorrect install.
  • npm is doing something I don’t understand.
  • Other ( see below for feature requests ):

What’s going wrong?

doc/files/npm-package-locks.md should be updated according to the changes introduced in npm 5.1.0 by means of #16866.

In that document it says:

The presence of a package lock changes the installation behaviour such that:

  1. The module tree described by the package lock is reproduced. This means reproducing the structure described in the file, using the specific files referenced in “resolved” if available, falling back to normal package resolution using “version” if one isn’t.

This holds no longer true since npm 5.1.0, because now the generated module tree is a combined result of both package.json and package-lock.json . (Example: package.json specifies some package with version ^1.1.0 ; package-lock.json had locked it with version 1.1.4 ; but actually, the package is already available with version 1.1.9 . In this case npm i resolves the package to 1.1.9 and overwrites the lockfile accordingly, hence ignoring the information in the lock file.)

  1. The documentation should be corrected and clarified, so that users can understand the behaviour of the lock file. (Maybe other places are affected as well, that I don’t know of.)
  2. It would be helpful if the documentation would point out the preferred way to perform reproducible builds, as this is not obvious anymore. This is especially problematic, because there is already some confusion for people who are trying to achieve deterministic build behaviour, e.g. on CI platforms.

(Thomas Grainger) #4

@zkat can we get a response to this rather than moving it to #support ?


(Felix Becker) #5

This is a big problem for automatic dependency update tools like Renovate: https://github.com/renovatebot/renovate/issues/2294
Essentially the update is not fully automatic, because after every update, I have to fix the lockfile manually again.


(Thomas Grainger) #6

using npm ci isn’t really an option - because we want to detect cases where a developer changed package.json but didn’t generate the package-lock.json or generated it on a different platform with different optional deps eg MacOS


(Felix Becker) #7

I set up a minimal reproduction case in a GitHub repository: felixfbecker/npm-changing-lockfile-repro


(Artur Eshenbrener) #8

I can reproduce bug with changed optional: true flags event on single machine without change platform/os.

$ ls
package.json

$ npm install
$ ls
node_modules  package.json  package-lock.json

$ git add .
$ git commit "install"
$ rm -rf node_modules
$ npm install
$ git status
changed package-lock.json

$ git diff
# lot of added optional: true lines
# okay, let commit it

$ git commit -am "change in lockfile"
$ rm -rf node_modules
$ npm install
$ git status
changed package-lock.json
$ git diff
# lot of added optional: true, but in other packages
# okay, commit it again
$ git commit -am "lockfiles, again"
$ rm -rf node_modules
$ npm install
$ git status
no changes # finally

You can get my package.json dependencies here for local reproduce: https://gist.github.com/Strate/addb41e6db3ae299e59b2180a6684f28

UPD: also, sometimes “optional: true” lines gonna to be removed, but I didn’t caught actusl case.

UPD2: just make npm i somepackage removes “optional: true” from some packages.


(Artur Eshenbrener) #9

Is there any plans to fix it?


(Opack) #10

We do have the same problem here. I applied this fix and it helps, but there are still some machines that produce different package-lock.json files. The differences can be on:

  • some “optional = true” (dis)appearing
  • some “dev = true” (dis)appearing
  • some “https” becoming “http” (or the other way around).

Special note on the https <-> http. This one is actually strange because for example yargs 11.1.0 can be in HTTPS on some machine, which I don’t understand because when I read http://registry.npmjs.org/yargs/, I can see that the referenced tarball for this version is http and not https. I must be missing something here… And of course npm config get registry outputs https://registry.npmjs.org/

PS : we all have node 8.11.4 or 8.12, and NPM 6.4.1.


(Lars Willighagen) #11

I think we have some progress on the optional and dev part in this thread. That fix fixes at least one of the problems, I’m not sure if there are any other. As for HTTPS <-> HTTP, I believe that’s covered here.


(John Hunter) #13

I continue to see all 3 of these changes. I’m on node 10.12.0, npm 6.4.1

The fix to pin all dev environments to specific versions isn’t ideal. Would be good to get a resolution to this.

Thanks.


(Lars Willighagen) #14

The http/https issue is being tracked here:

And the optional/dev issue can be tracked here:


(Graham Fairweather) #15

We have run into these issues too. We develop on Mac and then AWS builds from GitHub and so see changes with the optional flag and sometimes the http/https changes too. (Node v8.11.4 and NPM v6.4.1) Our team would love to see a solution to these issues.