Package-lock.json critical lock mechanism broken


(Tony) #1

What I Wanted to Do

Replicate dependencies across devices

What Happened Instead

dep-lock automatically broken.
npm install does not respect the checked in dependency manifest and
liberally modifies the lock-file that should be considered immutable.

Reproduction Steps

$ git reset --hard &&  git status
nothing to commit, working tree clean

$ npm install && git status
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git checkout -- <file>..." to discard changes in working directory)

	modified:   package-lock.json

Details

It’s quite a absurd that I have to educate you guys on what the purpose of a lock-file is, but here it goes:

A dependency lock is an immutable manifest containing the total sum of all dependencies and their versions.
The point of having such a file is that It ensures that the exact dependencies that are verified and checked in - get replicated in their exact form to the next device at any cost, be it a production environment or another developer’s computer.

Failure to honour the lock means failure to reproduce an asserted environment - ultimately failing to to be fit for production.

Changes to a lock should only ever be done as a response to manual action such as installing a previously unknown dependency or manually upgrading a dependency.
Automatic changes are strictly forbidden.

If I have misinterpreted the purpose of package-lock.json file then please rename it to something more fitting as package-cache.json to avoid further confusion.

Here’s a duplicate of this issue that was carelessly closed and locked:

Platform Info

$ npm --versions
 npm: '6.4.1',
  ares: '1.15.0',
  cldr: '33.1',
  http_parser: '2.8.0',
  icu: '62.1',
  modules: '64',
  napi: '3',
  nghttp2: '1.34.0',
  node: '10.14.2',
  openssl: '1.1.0j',
  tz: '2018e',
  unicode: '11.0',
  uv: '1.23.2',
  v8: '6.8.275.32-node.45',
  zlib: '1.2.11' }

$ node -p process.platform
linux

(Kat Marchán) #2

Moved to support. I can’t use these reproduction steps. There’s possibly specific bugs that cause different platforms to change the lock, but that is not intended and should be fixed on an individual basis. Your understanding of the lock is generally our understanding as well, bugs notwithstanding


(Lars Willighagen) #3

If you want a completely frozen, clean install solely based on package-lock.json, check out npm clean-install.


The default behavior of npm install may change package-lock.json if the latter is deemed invalid or incomplete. Most of that is caused by previous versions of npm. For example,

However, as you can see from that last topic, npm does generally respect the dependency structure of the lock if it is valid, whether it’s ideal or not.

As @zkat said before I finished this post, without some more specific information we can’t determine if the change is expected or not.


(Tony) #4
git clone git@github.com:mafintosh/hyperdrive.git
npm install
git status

Lock is ignored.


(Tony) #6

So you’re basically saying that newer versions of npm are backwards incompatible and silently change your lock as a fallback. That’s quite a security issue; since you might end up with code on your sensitive environments that has not been audited nor reviewed.

May i suggest that if for any reason at all; npm cannot replicate the expected state; and is forced to break the lock; That it atleast notifies that developer as a call to manual action?
Silently breaking locks is not a qualitative behaviour.


(Kat Marchán) #7

Thanks for the specific example! I’ve looked into it, and there’s two more case where we modify the package lock:

  1. If you’re crossing the npm@5 -> npm@6 boundary, there’s a one-time, big diff from modifying (or adding) the requires field. This change was a compromise to make it possible for us to further stabilize the lockfile in some situations.
  2. If your lockfile was generated by a buggy version of npm and the entries are messed up – we’ll fix them.

Both of these happened in your case. Things should be more stable going forward.

As @larsgw mentioned, if you want an absolute guarantee without any of the human-oriented features that can modify the pkglock, use npm ci, which is guaranteed to never modify your lock file in any way. npm install will continue to do what I consider reasonable things to improve the overall UX of working with a lockfile. It used to modify it even less, but we found that it caused even bigger problems for users. Everything is a compromise, and the specific UX around lockfiles, modulo bugs that we will definitely fix, is what I believe the ideal UX to be.


(Tony) #8

Thank you for clarifying, but I as many others would appreciate if you don’t try and reinvent ‘UX’ of a lock-mechanizm using a generalization; A lock is a lock, if it’s not a lock then don’t call it a lock.

It’s pretty much the same if you would come home one day and find your front-door unlocked… and when you talk to the lock-company’s support they tell you it’s a new feature, “you door will unlock sometimes when it makes sense without telling you, but don’t be alarmed, it will most of the time stay locked unless of course it has a general reason to.”


(Kat Marchán) #9

A broken lockfile should be fixed because it won’t do what was intended at all. That’s one issue. Another issue is if you upgrade to a semver-major version of npm, it’s reasonable to make a change, I think, that will specifically be there to prevent future unnecessary modifications (the change was fixing a previous bug that we just couldn’t fix without that format modification).

The last intended way is if you modify your package.json, we will adjust the lockfile to match it – so your “lock mechanism” is actually your package-lock.json plus your package.json combined. The reason for this is because many, many users add dependencies to package.json manually or with third-party tools, and expected npm install to install those new dependencies (or remove lines that were removed). Originally, we didn’t do this, and it was WAY worse UX. That’s the main thing I’m referring to when I talk about UX.

Currently, a third dimension of npm semver-major version is added – going forward, I don’t believe what we did with npm@6 will be repeated so it’ll be those two files, and that’s it.

The rest is bugs, and we’ll fix them. I can’t promise we will never change formats again, but I believe the lockfile is way more stable now than before, and will only get more stable as we stamp out further issues. I confidently believe that once things are stamped out further and people are done crossing the npm@6 threshold, you’ll find the UX we end up with more than satisfactory, but sometimes things take time, and I appreciate you being patient and understanding as we get there.