Bug in registry cache invalidation?

What I Wanted to Do

Publish new versions of a set of packages. Have users be able to install the new versions.

What Happened Instead

I published new versions of the packages. They appeared on the npm website. I was able to find the version using npm view. When installing, users got a message saying the version did not exist.

Reproduction Steps

Not currently reproducible - seems to be some issues in the cache invalidation of registry.npmjs.com - see below.

Details

We use a monorepo setup (with lerna) and recently published new versions of 50 packages to npm. As explained above, the versions appeared on the npm website and using npm view, but when installing, the modules were not downloadable - the module did not exist.

Couldn’t find any versions for “@sanity/mutator” that matches “0.141.5”

Some of the published packages did appear, but they were referencing packages that did not appear, so it all broke down. The modules we generally saw fail were @sanity/client, @sanity/preview and @sanity/mutator - for all I know it could have been more modules that were failing, but these were the first to try to download or some such.

I enabled verbose mode npm install -ddd to find the URLs that npm was requesting, and saw that https://registry.npmjs.com/@sanity%2Fmutator was being hit. Using curl to retrieve the URL gave me the right answer - it had the newly published version.

After some more digging, I noticed that the requests were using an accept header that included application/vnd.npm.install-v1+json, so I added that to my curl request and boom:

curl -XGET -I -H'Accept: application/vnd.npm.install-v1+json; q=1.0, application/json; q=0.8, */*' https://registry.npmjs.com/@sanity%2fmutator
HTTP/2 200
date: Fri, 28 Jun 2019 14:44:59 GMT
content-type: application/vnd.npm.install-v1+json
content-length: 367487
set-cookie: __cfduid=<redacted>; expires=Sat, 27-Jun-20 14:44:59 GMT; path=/; domain=.registry.npmjs.com; HttpOnly
cf-cache-status: HIT
cache-control: public, max-age=300
cf-ray: 4ee0711f4b0842b9-OSL
accept-ranges: bytes
age: 7195
etag: "5db23af6c3ae7ec085865537b4ca5163"
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
last-modified: Fri, 28 Jun 2019 12:22:51 GMT
vary: Accept-Encoding
x-amz-meta-rev: 447-df57650299f8209f12102b2d588fffa9

As you can see, it has an Age header that is 7195, which is 2 hours. Adding a cache breaker query parameter gave me a fresh copy which does include the new version. Eventually, things settled down and the new version appeared, but this was perhaps 2-3 hours after the publish.

I’m not familiar with what kind of cache invalidation infrastructure npm is running, but I thought it was worth reporting, given this has happened to us multiple times.

On a semi-related note, the Vary header does not seem to include Accept - this could potentially lead to clients caching the wrong response if they actually honor this header. I assume you also have an s-maxage set internally that is being removed on the way out - otherwise this should never have been cached beyond the 300s provided max-age.

Platform Info

$ npm --versions
{ npm: '6.9.0',
  ares: '1.15.0',
  cldr: '33.1',
  http_parser: '2.8.0',
  icu: '62.1',
  modules: '64',
  napi: '3',
  nghttp2: '1.34.0',
  node: '10.15.3',
  openssl: '1.1.0j',
  tz: '2018e',
  unicode: '11.0',
  uv: '1.23.2',
  v8: '6.8.275.32-node.51',
  zlib: '1.2.11' }
$ node -p process.platform
darwin

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.