`npm cache add` fails to make package available for install

What I Wanted to Do

I wanted to populate the cache with tarballs for unpublished versions of a project for cross-dependency testing prior to publish.

I used npm cache add <tarball> --global to populate the cache with the contents of the tarball. Examining locally I can observe that the cache has new entries (although seeing what they are is difficult).

I then tried both adding the version to the package.json of another project and running npm install and specifically running npm install <packageName>@<versionProducedLocally>.

I would expect npm cache add <tarball> to enable at least npm install package@version --offline or npm install package@version --prefer-offline to be able to install that version without error.

What Happened Instead

88 silly fetchPackageMetaData error for ember-data@3.12.0-canary.0.a18bf27b0a01e98da97d1a84c42a384c100bc71d No matching version found for ember-data@3.12.0-canary.0.a18bf27b0a01e98da97d1a84c42a384c100bc71d

....


3023 verbose type version
3024 verbose stack ember-data: No matching version found for ember-data@3.12.0-canary.0.a18bf27b0a01e98da97d1a84c42a384c100bc71d
3024 verbose stack     at pickManifest (/usr/local/lib/node_modules/npm/node_modules/npm-pick-manifest/index.js:101:11)
3024 verbose stack     at fetchPackument.then.packument (/usr/local/lib/node_modules/npm/node_modules/pacote/lib/fetchers/registry/manifest.js:24:14)
3024 verbose stack     at tryCatcher (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/util.js:16:23)
3024 verbose stack     at Promise._settlePromiseFromHandler (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:512:31)
3024 verbose stack     at Promise._settlePromise (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:569:18)
3024 verbose stack     at Promise._settlePromiseCtx (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/promise.js:606:10)
3024 verbose stack     at _drainQueueStep (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:142:12)
3024 verbose stack     at _drainQueue (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:131:9)
3024 verbose stack     at Async._drainQueues (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:147:5)
3024 verbose stack     at Immediate.Async.drainQueues [as _onImmediate] (/usr/local/lib/node_modules/npm/node_modules/bluebird/js/release/async.js:17:14)
3024 verbose stack     at processImmediate (internal/timers.js:443:21)
3025 verbose cwd /Users/cthoburn/Github/__external-test-cache/ember-m3
3026 verbose Darwin 18.2.0
3027 verbose argv "/usr/local/Cellar/node/11.13.0/bin/node" "/usr/local/bin/npm" "install" "--offline"
3028 verbose node v11.13.0
3029 verbose npm  v6.9.1-next.0
3030 error code ETARGET
3031 error notarget No matching version found for ember-data@3.12.0-canary.0.a18bf27b0a01e98da97d1a84c42a384c100bc71d
3032 error notarget In most cases you or one of your dependencies are requesting
3032 error notarget a package version that doesn't exist.
3032 error notarget
3032 error notarget It was specified as a dependency of 'ember-m3'
3033 verbose exit [ 1, true ]

Reproduction Steps

To setup a reproduction you may use the same app that led me to open this issue. You may also read up on the change that discovered this issue here: https://github.com/emberjs/data/pull/6128

Step 1: setup repo locally

git clone git@github.com:emberjs/data.git;
cd data;
git checkout e397d59f781eab8df46682f97bf038913c2e3ba8;
yarn install;

( yes yarn is required for the install as workspaces are involved, the rest of this is npm I swear <3)

Step 2: package up the commit and add to npm cache

This uses npm pack to produce tarballs and npm cache add <tarball> --global to add it to the cache.
It will output the versions produced.

node ./bin/packages-for-commit.js;

Step 3: try to use the inserted package version elsewhere

cd ..;
mkdir test-new-package;
cd test-new-package;
npm install ember-data@<version output from running node ./bin/packages-for-commit.js>

observe failure. For instance

npm install ember-data@3.12.0-canary.0.a18bf27b0a01e98da97d1a84c42a384c100bc71d.a18bf27b0a01e98da97d1a84c42a384c100bc71d
npm ERR! code ETARGET
npm ERR! notarget No matching version found for ember-data@3.12.0-canary.0.a18bf27b0a01e98da97d1a84c42a384c100bc71d.a18bf27b0a01e98da97d1a84c42a384c100bc71d
npm ERR! notarget In most cases you or one of your dependencies are requesting
npm ERR! notarget a package version that doesn't exist.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/cthoburn/.npm/_logs/2019-05-21T18_40_52_749Z-debug.log

Details

I am aware that often projects will utilize local registries for being able to locally publish and test packages in this way; however, we are looking to give consuming applications that we do not own the ability to quickly produce tarballs from a commit (usually a PR) to test a change against their own application. Having our consumers setup a registry to be able to use these generated tarballs seems like too much to ask.

Long term we will likely publish these to a custom registry hosted on heroku but not all consuming applications will be able to import those packages easily. For instance, an organization with their own registry and a firewall would find it difficult to be able to utilize tarballs hosted in a custom registry if the packages hosted there cross-linked dependencies with tarball urls pointed at our remote custom registry. We are looking for maximum shareability here.

A key component of why this approach was being looked at is that we are publishing a monorepo wherein packages that we publish may refer to other packages that we publish. For testing a consuming application will want to keep all of the packages that we publish “locked to the same commit”. Installing a single dependency via tarball url is much simpler and does not have this difficulty.

Platform Info

$ npm --versions

{ npm: '6.9.1-next.0',
  ares: '1.15.0',
  brotli: '1.0.7',
  cldr: '34.0',
  http_parser: '2.8.0',
  icu: '63.1',
  llhttp: '1.1.1',
  modules: '67',
  napi: '4',
  nghttp2: '1.34.0',
  node: '11.13.0',
  openssl: '1.1.1b',
  tz: '2018e',
  unicode: '11.0',
  uv: '1.27.0',
  v8: '7.0.276.38-node.18',
  zlib: '1.2.11' }

$ node -p process.platform

darwin