replicate.npmjs.com stream publishing really old events

registry
triaged

(Mike) #1

What I Wanted to Do

Listening to the http://replicate.npmjs.com/registry CouchDB stream to process new package versions being published.

What Happened Instead

It appears over the past couple of days tens of thousands of events have been added to the stream and many of them are out of order (based on modified dates) from extremely old packages. For example, here’s a recent event:

{ seq: 7348806,
  id: 'babel-plugin-wpm',
  changes: [ { rev: '2-f10f60a30cbbc209d606d19c853707a4' } ],
  doc: 
   { _id: 'babel-plugin-wpm',
     _rev: '2-f10f60a30cbbc209d606d19c853707a4',
     name: 'babel-plugin-wpm',
     description: 'Some specific transformers for Web Package Manager',
     'dist-tags': { latest: '0.0.2' },
     versions: { '0.0.1': [Object], '0.0.2': [Object] },
     readme: '# babel-plugin-wpm\n\nSome specific transformer(s) for [Web Package Manager](https://github.com/tnhu/wpm)\n\n## Input\n\n```js\nClass({\n  IndexRoute() {\n  },\n\n  signin() {\n  }\n});\n```\n\n## Output\n\n```js\nClass({\n  constructor: function IndexRoute() {\n  },\n\n  signin: function signin() {\n  }\n});\n```\n\n## Rules\n\n* If there is only one CamelCase method. The method becomes constructor.\n* If there are more than one CamelCase methods, the first CamelCase method becomes constructor.\n\n## Installation\n\n```sh\n$ npm install babel-plugin-wpm\n```\n\n## Usage\n\n### Via `.babelrc` (Recommended)\n\n**.babelrc**\n\n```json\n{\n  "plugins": ["babel-plugin-wpm"]\n}\n```\n\n### Via CLI\n\n```sh\n$ babel --plugins babel-plugin-wpm script.js\n```\n\n### Via Node API\n\n```javascript\nrequire("babel-core").transform("code", {\n  plugins: ["babel-plugin-wpm"]\n});\n```\n',
     maintainers: [ [Object] ],
     time: 
      { modified: '2015-07-04T15:04:24.043Z',
        created: '2015-07-04T07:27:56.302Z',
        '0.0.1': '2015-07-04T07:27:56.302Z',
        '0.0.2': '2015-07-04T15:04:24.043Z' },
     homepage: 'https://github.com/tnhu/babel-plugin-wpm',
     keywords: [ 'babel', 'babel-plugin', 'wpm' ],
     repository: 
      { type: 'git',
        url: 'git+https://github.com/tnhu/babel-plugin-wpm.git' },
     author: { name: 'Tan Nhu' },
     bugs: { url: 'https://github.com/tnhu/babel-plugin-wpm/issues' },
     license: 'MIT',
     readmeFilename: 'README.md' } }

These packages are extremely old and haven’t had any changes in years, yet are showing up under new sequence IDs despite the docs appearing to contain no changes from the previous (year’s old) versions. Are there any issues with the replicate stream at the moment that would explain this behavior? Did npm take any bulk operations that would have dumped all of these old packages back into the replicate stream as new events?

Reproduction Steps

Simply listen to the CouchDB replicate stream.

Details

N/A

Platform Info

N/A


(Chris Dickinson) #2

record scratch — Yep! That’s us. We’re currently backfilling https urls in versions.<versions>.dist.url for packages that previously pointed at http://. We’ve ratelimited this job to reduce load on our infrastructure, so these events will come in for a few more days. Sorry for the trouble!


Some packages have dist.tarball as http and not https
(Mike) #3

No worries - just wanted to make sure I wasn’t going crazy. Thanks for confirming!