npm install for package with local dependency fails

cli
help-wanted
priority:medium
triaged

(Jan Hecking) #1

What I Wanted to Do

Install a package that has local dependencies (i.e. “file:…”).

What Happened Instead

I get an error message that transitive dependency of the local dependency cannot be installed:

ENOENT: no such file or directory, rename ‘/XYZ/npm-install-local-dependency-bug/examples/node_modules/.staging/npm-install-local-dependency-bug-f307b53e/node_modules/acorn’ -> ‘/XYZ/npm-install-local-dependency-bug/examples/node_modules/.staging/acorn-af0fc6c4’

Reproduction Steps

$ git clone https://github.com/jhecking/npm-install-local-dependency-bug.git .
Cloning into '.'...
remote: Counting objects: 7, done.
remote: Compressing objects: 100% (5/5), done.
remote: Total 7 (delta 2), reused 7 (delta 2), pack-reused 0
Unpacking objects: 100% (7/7), done.

$ cat package.json
{
  "name": "npm-install-local-dependency-bug",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "standard": "^11.0"
  }
}

$ npm install
npm WARN npm-install-local-dependency-bug@1.0.0 No description
npm WARN npm-install-local-dependency-bug@1.0.0 No repository field.

added 214 packages from 173 contributors and audited 374 packages in 2.304s
found 0 vulnerabilities

$ cd examples/

$ cat package.json
{
  "name": "examples",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "author": "",
  "license": "ISC",
  "dependencies": {
    "npm-install-local-dependency-bug": "file:.."
  }
}

$ npm install
npm WARN examples@1.0.0 No description
npm WARN examples@1.0.0 No repository field.

npm ERR! path /XYZ/npm-install-local-dependency-bug/examples/node_modules/.staging/npm-install-local-dependency-bug-f307b53e/node_modules/acorn
npm ERR! code ENOENT
npm ERR! errno -2
npm ERR! syscall rename
npm ERR! enoent ENOENT: no such file or directory, rename '/XYZ/npm-install-local-dependency-bug/examples/node_modules/.staging/npm-install-local-dependency-bug-f307b53e/node_modules/acorn' -> '/XYZ/npm-install-local-dependency-bug/examples/node_modules/.staging/acorn-af0fc6c4'
npm ERR! enoent This is related to npm not being able to find a file.
npm ERR! enoent

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/jhecking/.npm/_logs/2018-07-20T04_01_29_488Z-debug.log

Details

2018-07-20T04_01_29_488Z-debug.log (21.0 KB)

Platform Info

$ npm --versions
{ examples: '1.0.0',
  npm: '6.2.0',
  ares: '1.14.0',
  cldr: '33.0',
  http_parser: '2.8.0',
  icu: '61.1',
  modules: '64',
  napi: '3',
  nghttp2: '1.32.0',
  node: '10.6.0',
  openssl: '1.1.0h',
  tz: '2018c',
  unicode: '10.0',
  uv: '1.21.0',
  v8: '6.7.288.46-node.13',
  zlib: '1.2.11' }
$ node -p process.platform
darwin

Installing dependencies fails when a file: dependency requires @babel/preset-env
Installing dependencies fails when a file: dependency requires @babel/preset-env
npm install does not install transitive dependencies of local dependency
(Kat Marchán) #2

Note: This seems to be a regression since npm@5.10.0-next.0. I can’t repro it with npm@5.9.0-next.0 or anything reasonably earlier.

I’d appreciate some help if someone was interested in looking at the changelog for 5.10.0-next.0 and see if anything jumps out at them. (I’m just triaging right now).


(Jan Hecking) #3

One more observation that might or might not be helpful to pin down the exact issue:

If I delete the package-lock.json file in the examples subdirectory, prior to running npm install in that directory, then the command succeeds.


(Jan Hecking) #4

From that changelog, this one entry stands out to me:

0dffa9c2a 609d6f6e1 08f81aa94 f8b76e076 6d609822d 59d080a22 Restore the ability to bundle dependencies that are uninstallable from the registry. This also eliminates needless registry lookups for bundled dependencies.

Fixed a bug where attempting to install a dependency that is bundled inside another module without reinstalling that module would result in ENOENT errors. (@iarna)

It has to do with bundled dependencies, which the acorn package is in this case. And because it’s supposed to address ENOENT errors, which is exactly the kind of error that is now occurring. Might be a red herring, off course.


(Eric Amodio) #5

I am seeing the same issue with npm v6.2.0


(Kat Marchán) #6

You wanna do a git bisect on that? :slight_smile:


(Jan Hecking) #7

Here is the result of running git bisect:

$ git bisect start v6.2.0 v5.9.0-next.0
Bisecting: a merge base must be tested
[b1d3b2b33b71beb202ff612b12f7da623152737a] doc: update changelog for npm@5.9.0

$ git bisect run ../bisect_test
running ../bisect_test
Bisecting: 114 revisions left to test after this (roughly 7 steps)
[89102c0d995c3d707ff2b56995a97a1610f8b532] shrinkwrap: Prefer computed resolved from dep tree
running ../bisect_test
Bisecting: 56 revisions left to test after this (roughly 6 steps)
[0573d91e57c068635a3ad4187b9792afd7b5e22f] cacache@11.0.1
running ../bisect_test
Bisecting: 28 revisions left to test after this (roughly 5 steps)
[ff6b31f775f532bb8748e8ef85911ffb35a8c646] meta: Update npm's lockfile to ranges from versions
running ../bisect_test
Bisecting: 14 revisions left to test after this (roughly 4 steps)
[f9de7ef3a1089ceb2610cd27bbd4b4bc2979c4de] version: Let version succeed when `package-lock.json` is gitignored
running ../bisect_test
Bisecting: 7 revisions left to test after this (roughly 3 steps)
[471ee1c5b58631fe2e936e32480f3f5ed6438536] install/deps: fix bug where optional status not saved to package-lock
running ../bisect_test
Bisecting: 3 revisions left to test after this (roughly 2 steps)
[0da38b7b4aff0464c60ad12e0253fd389efd5086] test: Ensure optional status is saved to lock file
running ../bisect_test
Bisecting: 1 revision left to test after this (roughly 1 step)
[9dea95e319169647bea967e732ae4c8212608f53] is-only-optional: Read optional status from shrinkwrap when we have to
running ../bisect_test
Bisecting: 0 revisions left to test after this (roughly 0 steps)
[3888d20517593095038caa3cff68d697ae8769a8] inflate-shrinkwrap: Synthetic link deps get the right realpath
running ../bisect_test
3888d20517593095038caa3cff68d697ae8769a8 is the first bad commit
commit 3888d20517593095038caa3cff68d697ae8769a8
Author: Rebecca Turner <me@re-becca.org>
Date:   Wed Apr 11 17:51:06 2018 -0700

    inflate-shrinkwrap: Synthetic link deps get the right realpath

:040000 040000 aa2c50ece1903e2ecfd4bae6b5d439f11381511d 0c429da2ebace27245b47bd0210f032223150de9 M	lib
bisect run success

I’m not going to pretend I understand what that commit does, so someone else will probably have to take it from here. :slight_smile:


(Jan Hecking) #8

Just adding a link to the “first bad commit” identified by git bisect:

Hopefully @iarna has some idea what that commit does and why it might cause this problem?


(Rebecca Turner) #9

I mean, I know what the commit does, ofc, but the error is interesting…

So this class of ENOENT error occurs when we think we have a bundled dependency and we’re trying to unbundle it before continuing the install.

One of the quirks of file: type dependencies where we’re linking to a location outside of the location they’re being installed to is that they bring their own dependency tree, much like bundled dependencies. That you’re getting this implies that it’s getting confused and improperly trying to treat a linked dependency like a bundled dependency. Though in this case it has to do with a combination of links, bundles and transitive dependencies, which is apparently how our test suite overlooked it.


(Rebecca Turner) #10

Ok, I believe I have a fix for this. Transitive dependencies of the linked module that had been kept at least one level deep in the linked module would result in having the fact they were (deep) inside a symlinks lost when that tree was being reconstituted from a a lock-file.

All that being said, I think we shouldn’t be storing the children if linked deps that exist outside of our own tree in our lockfile at all, so I’ve patched that as well. I’ll put this in canary soonish so you all can give it a go.


(Jan Hecking) #11

Thanks @iarna! I gave your iarna/enoent-on-link-up branch a try and it does fix the problem for me!


(Rebecca Turner) #12

Issue: npm install with local packages and symlinks - ENOENT ...
(John Johnson) #13

Any update on this issue (or above PR)? Would love for this to land. Let me know if I can help at all!


(JT Turner) #14

I opened another PR with tests for it as the old one seems to have stalled.


(Erik Erikson) #15

We’ve also found this in every published 6.x release. Ensuring package-lock.json isn’t present solves the issue but creates a challenge in our travis-ci builds.


(Alexey Subach) #16

@jhecking @eamodio @johnrjj @jtwebman @erikerikson Let’s vote for this issue to draw more attention from NPM team! They seem to have recently added voting functionality - it’s next to the title of the issue. Sorry for spamming :)


(Erik Erikson) #17

To reproduce using serverless-artillery we can run:

git clone --depth=50 --branch=refactor-modularize-faas https://github.com/Nordstrom/serverless-artillery.git Nordstrom/serverless-artillery
cd Nordstrom/serverless-artillery/
npm ci

Ran last against the commit 4c6e921b76eae4edf08698c8b565b4f663a129a0


(Erik Erikson) #18

Neither PR appears to resolve this, unless I’ve misunderstood how to use them. I’d rate the probability of my misunderstanding to be ~0.7. I cloned the repo, switched to the branch, and ran node .../npm/cli ci.