npm install tgz: invalid config key requested

priority:medium
triaged
cli
help-wanted

(Clemens Buchacher) #1

What I Wanted to Do

Run npm install while package.json contains a file:path/to/tgz dependency.

What Happened Instead

Unhandled rejection Error: invalid config key requested: errors

Reproduction Steps

$ docker run --rm -it node:10.11.0-stretch sh -c 'git clone -q https://github.com/cbuchacher/npm-confkey-test && cd npm-confkey-test && ./test.sh'
+ npm pack file:lib
npm notice
npm notice package: lib@0.0.0
npm notice === Tarball Contents ===
npm notice 42B package.json
npm notice === Tarball Details ===
npm notice name:          lib
npm notice version:       0.0.0
npm notice filename:      lib-0.0.0.tgz
npm notice package size:  141 B
npm notice unpacked size: 42 B
npm notice shasum:        29770fb2767e8a375f4a2db57c2ca6becc0eccab
npm notice integrity:     sha512-P9SpPa2p/8Wsn[...]kVFJu8epKoynw==
npm notice total files:   1
npm notice
lib-0.0.0.tgz
+ npm install
Unhandled rejection Error: invalid config key requested: errors
    at BadKeyError (/usr/local/lib/node_modules/npm/node_modules/figgy-pudding/index.js:93:23)
    at pudGet (/usr/local/lib/node_modules/npm/node_modules/figgy-pudding/index.js:101:5)
    at FiggyPudding.get (/usr/local/lib/node_modules/npm/node_modules/figgy-pudding/index.js:27:12)
    at Object.get (/usr/local/lib/node_modules/npm/node_modules/figgy-pudding/index.js:159:16)
    at Object.checkData (/usr/local/lib/node_modules/npm/node_modules/ssri/index.js:232:22)
    at write (/usr/local/lib/node_modules/npm/node_modules/cacache/lib/content/write.js:34:31)
    at putData (/usr/local/lib/node_modules/npm/node_modules/cacache/put.js:29:10)
    at Object.x.put (/usr/local/lib/node_modules/npm/node_modules/cacache/locales/en.js:28:37)
    at readFileAsync.then.data (/usr/local/lib/node_modules/npm/node_modules/pacote/lib/fetchers/file.js:38:28)
    at tryCatcher (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:638:18)
    at /usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/nodeback.js:42:21

npm ERR! cb() never called!

npm ERR! This is an error with npm itself. Please report this error at:
npm ERR!     <https://npm.community>

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2018-09-26T11_15_15_882Z-debug.log

Details

I do not have the issue on Windows.

A related issue was fixed in 6.3.0, but I am using 6.4.1.

root@33e96b40f35e:/npm-confkey-test# cat /root/.npm/_logs/2018-09-26T11_20_16_650Z-debug.log
0 info it worked if it ends with ok
1 verbose cli [ '/usr/local/bin/node', '/usr/local/bin/npm', 'install' ]
2 info using npm@6.4.1
3 info using node@v10.11.0
4 verbose npm-session d271eab7ca939913
5 silly install runPreinstallTopLevelLifecycles
6 silly preinstall npm-confkey-test@0.0.0
7 info lifecycle npm-confkey-test@0.0.0~preinstall: npm-confkey-test@0.0.0
8 silly install loadCurrentTree
9 silly install readLocalPackageData
10 timing stage:loadCurrentTree Completed in 7ms
11 silly install loadIdealTree
12 silly install cloneCurrentTreeToIdealTree
13 timing stage:loadIdealTree:cloneCurrentTree Completed in 1ms
14 silly install loadShrinkwrap
15 timing stage:loadIdealTree:loadShrinkwrap Completed in 4ms
16 silly install loadAllDepsIntoIdealTree
17 timing stage:loadIdealTree:loadAllDepsIntoIdealTree Completed in 3ms
18 timing stage:loadIdealTree Completed in 9ms
19 silly currentTree npm-confkey-test@0.0.0
20 silly idealTree npm-confkey-test@0.0.0
20 silly idealTree `-- lib@file:lib-0.0.0.tgz
21 silly install generateActionsToTake
22 timing stage:generateActionsToTake Completed in 4ms
23 silly diffTrees action count 1
24 silly diffTrees add lib@file:lib-0.0.0.tgz
25 silly decomposeActions action count 8
26 silly decomposeActions fetch lib@file:lib-0.0.0.tgz
27 silly decomposeActions extract lib@file:lib-0.0.0.tgz
28 silly decomposeActions preinstall lib@file:lib-0.0.0.tgz
29 silly decomposeActions build lib@file:lib-0.0.0.tgz
30 silly decomposeActions install lib@file:lib-0.0.0.tgz
31 silly decomposeActions postinstall lib@file:lib-0.0.0.tgz
32 silly decomposeActions finalize lib@file:lib-0.0.0.tgz
33 silly decomposeActions refresh-package-json lib@file:lib-0.0.0.tgz
34 silly install executeActions
35 silly doSerial global-install 8
36 verbose correctMkdir /root/.npm/_locks correctMkdir not in flight; initializing
37 verbose makeDirectory /root/.npm/_locks creation not in flight; initializing
38 silly makeDirectory /root/.npm/_locks uid: 0 gid: 0
39 timing audit compress Completed in 5ms
40 info audit Submitting payload of 361bytes
41 verbose lock using /root/.npm/_locks/staging-0015357eff3ebaa7.lock for /npm-confkey-test/node_modules/.staging
42 silly doParallel extract 1
43 silly extract lib@file:lib-0.0.0.tgz
44 silly tarball trying lib@file:lib-0.0.0.tgz by hash: sha512-4ftF4ik7Al7+5B0CchmVUs8GDXp112YlM1UxZItQZz+/vs6j57YIzbQyXJDFxCx9GdjkXmKGsMqH2nlq6gERCA==
45 silly tarball no local data for lib@file:lib-0.0.0.tgz. Extracting by manifest.
46 timing audit submit Completed in 1046ms
47 http fetch POST 200 https://registry.npmjs.org/-/npm/v1/security/audits/quick 947ms
48 timing audit body Completed in 1ms
49 timing npm Completed in 1350ms
50 error cb() never called!
51 error This is an error with npm itself. Please report this error at:
52 error <https://npm.community>

Platform Info

$ npm --versions
{ 'npm-confkey-test': '0.0.0',
  npm: '6.4.1',
  ares: '1.14.0',
  cldr: '33.1',
  http_parser: '2.8.0',
  icu: '62.1',
  modules: '64',
  napi: '3',
  nghttp2: '1.33.0',
  node: '10.11.0',
  openssl: '1.1.0i',
  tz: '2018e',
  unicode: '11.0',
  uv: '1.23.0',
  v8: '6.8.275.32-node.28',
  zlib: '1.2.11' }

(Lars Willighagen) #2

The invalid key is actually just error, the s is left from the progress bar. Due to this line, probably:

However, implementing the same ‘fix’ as in that related issue brings me to a different error:

Unhandled rejection Error: Integrity check failed:
  Wanted: sha512-4ftF4ik7Al7+5B0CchmVUs8GDXp112YlM1UxZItQZz+/vs6j57YIzbQyXJDFxCx9GdjkXmKGsMqH2nlq6gERCA==
   Found: sha512-P9SpPa2p/8WsnVrZRpC8s3qbh5o9B4p432pBI/L9XoP4jrFwKstIdm2lKvT0tCGB7FE797OPykVFJu8epKoynw==
    at checksumError (/.../npm/node_modules/cacache/lib/content/write.js:157:13)
    at write (/.../npm/node_modules/cacache/lib/content/write.js:35:22)
    at putData (/.../npm/node_modules/cacache/put.js:30:10)
    at Object.x.put (/.../npm/node_modules/cacache/locales/en.js:28:37)
    at readFileAsync.then.data (/.../npm/node_modules/pacote/lib/fetchers/file.js:38:28)
    at tryCatcher (/.../npm/node_modules/bluebird/js/release/util.js:16:23)
    at Promise._settlePromiseFromHandler (/.../npm/node_modules/bluebird/js/release/promise.js:512:31)
    at Promise._settlePromise (/.../npm/node_modules/bluebird/js/release/promise.js:569:18)
    at Promise._settlePromise0 (/.../npm/node_modules/bluebird/js/release/promise.js:614:10)
    at Promise._settlePromises (/.../npm/node_modules/bluebird/js/release/promise.js:693:18)
    at Promise._fulfill (/.../npm/node_modules/bluebird/js/release/promise.js:638:18)
    at /.../npm/node_modules/bluebird/js/release/nodeback.js:42:21
    at /.../npm/node_modules/graceful-fs/graceful-fs.js:78:16
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:511:3)

Edit: so the “Found” integrity seems to be equal to the pack results, and the “Wanted” is coming from pacote - but I don’t know where pacote gets it from.


(Lars Willighagen) #3

Okay, that last bit is just caused by and outdated package-lock.json (would you believe I have been debugging that integrity for an entire hour!). I’ve made a PR:


(Clemens Buchacher) #4

Thanks for the fix.

There are two reasons for the changed hash:

  1. The file modes for package.json were different (666 and 644).
  2. The gzip header contains an OS specific code (0x3 on Linux, 0xa on Windows): https://github.com/nodejs/node/blob/master/deps/zlib/deflate.c#L867

The first one can be fixed by the packager, but the OS code can be overwritten only by setting a custom gzip header with deflateSetHeader. This API is not currently exposed in https://github.com/nodejs/node/blob/master/src/node_zlib.cc. After normalizing the timestamps https://github.com/npm/npm/issues/17412, it would be nice if we could also make the packages OS independent.