Some packages have dist.tarball as http and not https

registry

(Karan Thakkar) #1

What I Wanted to Do

Generate a pkglock with no packages having an http resolved url

What Happened Instead

Some packages in the pkglock had http urls. Specifically, these two packages (onetime@1.1.0 and compression@1.7.2) have their dist.tarball fields as http instead of https.

Reproduction Steps

Use the package.json in this gist to generate the pkglock: https://gist.github.com/karanjthakkar/6049a743c1d3d57bb829bde87ea851a9

Details

  1. Versions:
{ npm: '5.6.0',
  ares: '1.10.1-DEV',
  cldr: '32.0',
  http_parser: '2.8.0',
  icu: '60.1',
  modules: '57',
  napi: '3',
  nghttp2: '1.32.0',
  node: '8.11.3',
  openssl: '1.0.2o',
  tz: '2017c',
  unicode: '10.0',
  uv: '1.19.1',
  v8: '6.2.414.54',
  zlib: '1.2.11' }
  1. npm info onetime@1.1.0 dist.tarball gives http://registry.npmjs.org/onetime/-/onetime-1.1.0.tgz. npm info compression@1.7.2 dist.tarball gives http://registry.npmjs.org/compression/-/compression-1.7.2.tgz.

npm install downgrading resolved packages from https to http registry in package-lock.json
package-lock.json keeps changing between platforms and runs
npm install downgrading resolved packages from https to http registry in package-lock.json
(Kat Marchán) #2

npm view info for both of these packages:

# onetime
➜ npm view onetime@1.1.0 dist.tarball
http://registry.npmjs.org/onetime/-/onetime-1.1.0.tgz
➜ npm view onetime@latest dist.tarball
https://registry.npmjs.org/onetime/-/onetime-2.0.1.tgz

# compression
➜ npm view compression@1.7.2 dist.tarball # this is also latest
http://registry.npmjs.org/compression/-/compression-1.7.2.tgz

It looks pretty isolated to specific packages at some specific versions. Might be worthwhile to throw a follower at the registry and find any others with weird data like this.


(Kat Marchán) #3

As far as making sure you’re fetching these from https goes: because of the way the main registry works, you should be able to replace the http with https and it’ll Just Work™. I’ve forwarded this to the @services-team in the meantime so they can take a look and see if there’s any others. Not sure why this happened.


(Karan Thakkar) #4

Perfect! That’s what we have right now: a precommit hook to replace http with https. :slight_smile:


(Karan Thakkar) #5

Hello, just wanted to provide more info that I came across today. In a previous version of our pkglock, I can see that the dist.tarball for onetime@1.1.0 is https://registry.npmjs.org/onetime/-/onetime-1.1.0.tgz. So maybe something happened in the registry at some point in time that it started returning http urls instead of the old https ones.


(Kat Marchán) #6

yeah we already figured out what it is and know how to fix it. It turns out to have affected certain package versions of many packages, so we need to run a pretty big job before it’s fixed. It only happened for a little while, and it was pretty easy to scan for. Hang tight and it’ll be fixed eventually.


(Jeremy Thomerson) #7

@zkat what’s the status here? This is a pretty significant problem … it’s a bit shocking to see so many issues around the internet for this problem and it not getting the attention it deserves.


(Andrew Goode) #8

Is this the cause of the issue here? npm install downgrading resolved packages from https to http registry in package-lock.json

I’m still seeing specific versions of packages resolving to http, so I assume it’s still an issue. Any updates?

Here are some examples:

$ npm info minimist dist.tarball
http://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz

$ npm info stream-combiner dist.tarball
http://registry.npmjs.org/stream-combiner/-/stream-combiner-0.2.2.tgz

$ npm info got@6.7.1 dist.tarball
http://registry.npmjs.org/got/-/got-6.7.1.tgz

$ npm info pause-stream dist.tarball
http://registry.npmjs.org/pause-stream/-/pause-stream-0.0.11.tgz

(Bjorn Stromberg) #9

@services-team Can we get an update on this? It’s been over two months and we’re still getting http instead of https in our tarball URIs.

$ npm info create-hash dist.tarball
http://registry.npmjs.org/create-hash/-/create-hash-1.2.0.tgz

 $ npm info browserify-rsa dist.tarball
http://registry.npmjs.org/browserify-rsa/-/browserify-rsa-4.0.1.tgz

(Julien Vanier) #10

My projects are also affected by some packages returning http registry links.

$ npm show onetime@1 dist.tarball
onetime@1.0.0 'http://registry.npmjs.org/onetime/-/onetime-1.0.0.tgz'
onetime@1.1.0 'http://registry.npmjs.org/onetime/-/onetime-1.1.0.tgz'

$ npm show lodash@4.17 dist.tarball
lodash@4.17.0 'http://registry.npmjs.org/lodash/-/lodash-4.17.0.tgz'
lodash@4.17.1 'http://registry.npmjs.org/lodash/-/lodash-4.17.1.tgz'
lodash@4.17.2 'http://registry.npmjs.org/lodash/-/lodash-4.17.2.tgz'
lodash@4.17.3 'http://registry.npmjs.org/lodash/-/lodash-4.17.3.tgz'
lodash@4.17.4 'http://registry.npmjs.org/lodash/-/lodash-4.17.4.tgz'
lodash@4.17.5 'https://registry.npmjs.org/lodash/-/lodash-4.17.5.tgz'
lodash@4.17.9 'https://registry.npmjs.org/lodash/-/lodash-4.17.9.tgz'
lodash@4.17.10 'https://registry.npmjs.org/lodash/-/lodash-4.17.10.tgz'
lodash@4.17.11 'https://registry.npmjs.org/lodash/-/lodash-4.17.11.tgz'

(Michael Hart) #11

@services-team Would love an update too – this is affecting us on a daily basis on a whole range of packages (buffer, commander, minimist, mkdirp, sax). Is the job to fix it still running or has it raised its head again?


(Bjorn Stromberg) #12

@ceejbot @isaacs Can someone chime in here? Is there a good reason to still be serving npm packages over http?


(Jeremy Thomerson) #13

@zkat @services-team this is a significant issue for many users, yet you’re providing no updates on what’s happening. It’s been just shy of three months since you said you knew what the problem was and needed to run a job to update the links. I understand that you’re providing this service free of charge, but that’s part of the goal of your project, so the community at least deserves some transparency on why it’s taking months to update the database (or data store of some type) to return https links instead of http. Thank you in advance for any information you can provide.


(Kat Marchán) #14

it’s still being worked on, don’t worry. Just wait :slight_smile:

If it’s super important to make sure you have pkglocks that only do https, just use sed -i and replace them accordingly. They’re still hosted behind https.

Beyond that, I’ve no timeline or info besides the fact that it’s being tracked internally.


(Francis Gulotta) #15

Is there a npm script we can use to correct the package-lock.json after install? The security is important (unsure how the the sha1is used) but constant thrashing of the lockfile is a pain.

I’ve tried

{
  "scripts": {
    "install": "sed -i '' -e 's/http:\\/\\//https:\\/\\//g' package-lock.json",
  }
}

but it doesn’t work, the lock file seems to be written afterwards


(Adam Baldwin) #16

Just for some communication here from the security team. While I initially escalated this internally to our engineering team and talked with the cli team I’ve learned some new information that I wanted to communicate.

We’re going to fix this on the registry side, however the risk is reduced based on some information the cli team provided me; The npm cli will always use HTTPS when talking to the registry for tarballs even though it says HTTP.

This is demonstrated by using ``–loglevel=http` and looking at the url that the tarballs are pulled from.

I should note that users should make sure that their .npmrc files do not reference a http:// registry

I’m also working internally to understand the reasoning behind why we have HTTP enabled for the npm Registry and what it would take and what impact it would have should we disable it.

I’ll provide more information on timing, etc when I have it.


(Bjorn Stromberg) #17

If the npm cli will always use HTTPS, why can’t the lock file reflect that?


(Kurt Miller) #18

This is working for me for both Mac and Linux:

{
  "scripts": {
    "postshrinkwrap": "if [ \"`uname`\" = \"Darwin\" ]; then sed -i '' -e 's/http:\\/\\//https:\\/\\//g' package-lock.json; else sed -i -e 's/http:\\/\\//https:\\/\\//g' package-lock.json; fi"
  }
}

(Adam Baldwin) #19

@bjornstar It’s very possible and as I mentioned we’re going to fix this and make sure that dist urls will represent HTTPS vs HTTP.