npm Community Forum (Archive)

The npm community forum has been discontinued.

To discuss usage of npm, visit the GitHub Support Community.

Some packages have dist.tarball as http and not https

Just for some communication here from the security team. While I initially escalated this internally to our engineering team and talked with the cli team I’ve learned some new information that I wanted to communicate.

We’re going to fix this on the registry side, however the risk is reduced based on some information the cli team provided me; The npm cli will always use HTTPS when talking to the registry for tarballs even though it says HTTP.

This is demonstrated by using ``–loglevel=http` and looking at the url that the tarballs are pulled from.

I should note that users should make sure that their .npmrc files do not reference a http:// registry

I’m also working internally to understand the reasoning behind why we have HTTP enabled for the npm Registry and what it would take and what impact it would have should we disable it.

I’ll provide more information on timing, etc when I have it.


Nope, that would completely ignore the package-lock file during install and you would end up with the most recent packages. More or less the same thing that happens when one removes the package-lock file.


Wow, @coderkevin! I’m so glad I had this issue 2 hours after you commented. That saved me a lot of time, thanks!


using native find command
"postshrinkwrap": "find . -name node_modules -prune -o -name package-lock.json -exec sed -i 's/http:\/\//https:\/\//g' {} +"


Great suggestion Kevin, thank you!


Removing node_modules directory, mentioned here, helps me to resolve the problem


@services-team Would love an update too – this is affecting us on a daily basis on a whole range of packages (buffer, commander, minimist, mkdirp, sax). Is the job to fix it still running or has it raised its head again?


Did you get any workaround for this issue? We are in a similar situation & any inputs is greatly appreciated.


I can confirm this happens on npm 6.4.1 on windows


Note that modern versions of the CLI can automatically resolve git conflicts on the pkglock for you with npm i --package-lock-only. Additionally, you can replace all instances of http: with https: by hand and have it work just fine.


@zkat I love all of the work being done on npm, great improvements.

Yet 3 months since your comment that the fix would be easy and would roll out eventually to everyone, we’re still stuck with flipflopping http & https in package-lock.json files. This happens between developers that have identical versions of node & npm installed.

Is there a proposed resolution already? (Maybe we need to clear npm local machine caches?)

While I love npx, this issue is something that hits our teams on a day to day basis, while I only use npx once every month, at maximum. May I ask what the priorities are on these issues that hit nearly all developers that work together with other developers on projects?


This is pretty low priority because it’s largely an aesthetic issue. You’re not making requests to http unless your npm config get registry says http:. I have no ETA for you.


Just remove package-lock.json from VCS.
It causes more trouble than it solves.

If someone insists to have a lock file, just dd if=/dev/urandom of=package-lock.json. npm does not generate it deterministically anyway.


Hi everyone, we’re discussing this internally so I want to give you a summary.

The changing http/https urls you’re seeing recorded come from inconsistent data in the registry. I don’t have an ETA on that fix, but someone else will chime in if they can. It’s important to note that the actual request for that url is being converted to https when appropriate, so this is only an issue of the data string being recorded to your package-lock.json.

The CLI team would like to figure out a way we can smooth this over so you only see the http/https url prefix you had in the file previously – the behavior on the backend will remain consistent. If we can figure out a straightforward way to accomplish this, a community member could submit a PR.

We’ll update when we have more. Thanks for your patience!


If the npm cli will always use HTTPS, why can’t the lock file reflect that?


As far as making sure you’re fetching these from https goes: because of the way the main registry works, you should be able to replace the http with https and it’ll Just Work™. I’ve forwarded this to the @services-team in the meantime so they can take a look and see if there’s any others. Not sure why this happened.


My projects are also affected by some packages returning http registry links.

$ npm show onetime@1 dist.tarball
onetime@1.0.0 'http://registry.npmjs.org/onetime/-/onetime-1.0.0.tgz'
onetime@1.1.0 'http://registry.npmjs.org/onetime/-/onetime-1.1.0.tgz'

$ npm show lodash@4.17 dist.tarball
lodash@4.17.0 'http://registry.npmjs.org/lodash/-/lodash-4.17.0.tgz'
lodash@4.17.1 'http://registry.npmjs.org/lodash/-/lodash-4.17.1.tgz'
lodash@4.17.2 'http://registry.npmjs.org/lodash/-/lodash-4.17.2.tgz'
lodash@4.17.3 'http://registry.npmjs.org/lodash/-/lodash-4.17.3.tgz'
lodash@4.17.4 'http://registry.npmjs.org/lodash/-/lodash-4.17.4.tgz'
lodash@4.17.5 'https://registry.npmjs.org/lodash/-/lodash-4.17.5.tgz'
lodash@4.17.9 'https://registry.npmjs.org/lodash/-/lodash-4.17.9.tgz'
lodash@4.17.10 'https://registry.npmjs.org/lodash/-/lodash-4.17.10.tgz'
lodash@4.17.11 'https://registry.npmjs.org/lodash/-/lodash-4.17.11.tgz'


@ceejbot @isaacs Can someone chime in here? Is there a good reason to still be serving npm packages over http?


@services-team Can we get an update on this? It’s been over two months and we’re still getting http instead of https in our tarball URIs.

$ npm info create-hash dist.tarball
http://registry.npmjs.org/create-hash/-/create-hash-1.2.0.tgz

 $ npm info browserify-rsa dist.tarball
http://registry.npmjs.org/browserify-rsa/-/browserify-rsa-4.0.1.tgz


Doing this seems to finally work:

npm install --global npm@6.6.0-next.0
rm -rf ./node_modules
npm cache clean --force
npm install --prefer-online

Thank you.


When I run the same command on my mac, save version of npm and node, it stays https …

** edit nevermind, the problem is still there on MacOS npm version 6.4.1

there are also changes to the dev and optional properties… not sure what they are for.


Remove the node_modules/ folder, package-lock.json and run ‘npm install --prefer-online’ again, it worked for me.


should I be using this on every install now?


I am having the same issue comparing package-lock.json files between mac os and linux origin.


Hey all, the rewrite job that was fixing this issue finished running. I recommend you fix any remaining issues you have by hand, and it should not repeat itself going forward (barring cached versions – use --prefer-online to make sure you’re not getting anything stale).

So I consider this issue resolved now. We likely won’t be making CLI-side changes related to this, so we should be all set.


This is definitely not just an aesthetic issue. This can and does lead to lost productivity in discerning real changes to package-lock.json vs the superfluous http-https flip-flopping when reviewing PRs for instance.


Please remove node_modules as well (along with doing everything else).

If this continues, please tell me what specific packages you experienced this with and I’ll take a look.


Everything is complete. Please note that npm will still download over https if your configured registry is https.



yeah we already figured out what it is and know how to fix it. It turns out to have affected certain package versions of many packages, so we need to run a pretty big job before it’s fixed. It only happened for a little while, and it was pretty easy to scan for. Hang tight and it’ll be fixed eventually.


Same here, @johnhunter; I’m wondering if this is a different issue? The primary issue above seems to be deterministic; that is, certain packages should, apparently, always be either http or https? But in our team, we have constant diff-noise of this form:

 "minimist": {
   "version": "1.2.0",
-  "resolved": "http://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
+  "resolved": "https://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz",
   "integrity": "sha1-o1AIsg9BOD7sH7kU9M1d95omQoQ="
},

Note that the version-number nor integrity have changed; this is the same version.

(Was there a recent change to this behaviour? One of the clients downgrading these same packages to http is at npm 6.3.0; one constantly committing re-upgrades to https is at npm 6.4.1.)


I really like @larsgw’s idea. As internally https is always used, it would be great if npm would just always output https urls, no matter what the registry returns. I know this is usually not npm’s responsibility as it can work with multiple registries. But this would release the burden from the registry team, and allow them to solve it within their own timeframe

On the other hand, I’m still confused why different machines get different urls from the registry. Load balanced machines/endpoints that serve different urls from corrupted caches? Registry responses that are stored locally in the npm cache on the machine?

And @zkat: Thanks for pointing me to --package-lock-only, that will allow us to solve this for our juniors! This is exactly the other way around, it only updates the package-lock. But I want to do an install without updating the package-lock, essentially npm ci, without removing the node_modules folder. But that does not seem to exist. Also, npm ci is not an option each time we start our project because we have a huge set of dependencies: over 20 secs of installation when removing the node_modules folder.


Is this the cause of the issue here? npm install downgrading resolved packages from https to http registry in package-lock.json

I’m still seeing specific versions of packages resolving to http, so I assume it’s still an issue. Any updates?

Here are some examples:

$ npm info minimist dist.tarball
http://registry.npmjs.org/minimist/-/minimist-1.2.0.tgz

$ npm info stream-combiner dist.tarball
http://registry.npmjs.org/stream-combiner/-/stream-combiner-0.2.2.tgz

$ npm info got@6.7.1 dist.tarball
http://registry.npmjs.org/got/-/got-6.7.1.tgz

$ npm info pause-stream dist.tarball
http://registry.npmjs.org/pause-stream/-/pause-stream-0.0.11.tgz


Removing it is not an option, I rely on npm ci to make sure I’m shipping exact versions of dependencies, I’m not sure why you are suggesting this.

Thanks.


While there’s no security risks regarding https or https, I wouldn’t say this is just an aesthetic issue. Our team alone spends multiple hours a week working around this bug. I have not seen any workaround that allows our developers to ignore the package-lock files. Junior developers are tripped up by this, and we have to explain them that this is a bug in npm, and that they should ignore these changes. Juniors easily commit these files, and this bug puts an extra burden on them and their PR reviewers / seniors.

Low priority bugs like this one (that keep on hanging around) are the exact reason yarn was able to gain so much popularity in a short time. I’m all for npm as the standard, but please reconsider the importance of tiny bugs that hit developers on a day to day basis.


Hello, just wanted to provide more info that I came across today. In a previous version of our pkglock, I can see that the dist.tarball for onetime@1.1.0 is https://registry.npmjs.org/onetime/-/onetime-1.1.0.tgz. So maybe something happened in the registry at some point in time that it started returning http urls instead of the old https ones.


@bjornstar It’s very possible and as I mentioned we’re going to fix this and make sure that dist urls will represent HTTPS vs HTTP.


Does --no-package-lock work (I believe that’s how Boolean flags work)?


Is this backfilling complete? In case I’m behind a corporate firewall, will this backfill help me in downloading any component that return a http url.


I wish npm would go back to github issues like everyone else… googling issues gets you to the npm/npm repo and then you have to search for hours in here… the bug discussion don’t show if a resolution has been found or not, I don’t like the experience…

so has this been fixed? this issues has been opened since May 2018 https://github.com/npm/npm/issues/20719

thank you…


I want to update with a case, and ask for a workaround since it is aesthetic. If there is a functionality lose you can only conisider it as minor if there is a strict workaround.

Here is the details:

We are using nexus as a corporate repository and http protocol is disabled. So considering https support of npm we enable proxy without direct access. But due to invalid entries in npm metadata nexus cannot download tarballs, we cannot have even initial build to have package-lock.json

Regards.


I’m behind a corporate firewall that will block tarball downloads over http. If a package (or its dependancy) has a http url in its metadata, it will be blocked and I not be able to install that package. I’m unable to use the package.lock.json trick as the package(s) have never been installed before.

Is there any other workaround for this issue while we wait for the fix ?


Is there a npm script we can use to correct the package-lock.json after install? The security is important (unsure how the the sha1is used) but constant thrashing of the lockfile is a pain.

I’ve tried

{
  "scripts": {
    "install": "sed -i '' -e 's/http:\\/\\//https:\\/\\//g' package-lock.json",
  }
}

but it doesn’t work, the lock file seems to be written afterwards


This is working for me for both Mac and Linux:

{
  "scripts": {
    "postshrinkwrap": "if [ \"`uname`\" = \"Darwin\" ]; then sed -i '' -e 's/http:\\/\\//https:\\/\\//g' package-lock.json; else sed -i -e 's/http:\\/\\//https:\\/\\//g' package-lock.json; fi"
  }
}


Any indication when this is going to be resolved? Manually adjusting the lock file (or using a script to do it) is not a long term solution. With contributors on different npm versions we get constant flip-flopping on the committed lock file.


If anyone’s interested, I made a workaround by adding the following to my package.json scripts (same idea as above, but works on all platforms)

	     "postshrinkwrap": "replace --silent 'http://' 'https://' ./package-lock.json",

Make sure to npm install --save-dev replace


No, just if you’re still seeing the issue


???

it’s creating git conflicts between Mac users, Windows users and sometimes CI, whenever a npm install is run

why would this be aesthetic? this is a productivity issue


I’ve got an idea:

https://github.com/zkat/pacote/pull/167


@zkat @services-team this is a significant issue for many users, yet you’re providing no updates on what’s happening. It’s been just shy of three months since you said you knew what the problem was and needed to run a job to update the links. I understand that you’re providing this service free of charge, but that’s part of the goal of your project, so the community at least deserves some transparency on why it’s taking months to update the database (or data store of some type) to return https links instead of http. Thank you in advance for any information you can provide.


This may be considered low priority at npm, but I agree with Phil - it’s really irritating for users. Churn on package-lock files every time someone runs npm i causes unnecessary noise on commits unless developers are careful and fix the http / https changes. I’d ask for you to reconsider the priority on this and get it escalated. Even as a low priority issue, for it to be affecting the community for more than 6 months doesn’t seem great.


npm view info for both of these packages:

# onetime
➜ npm view onetime@1.1.0 dist.tarball
http://registry.npmjs.org/onetime/-/onetime-1.1.0.tgz
➜ npm view onetime@latest dist.tarball
https://registry.npmjs.org/onetime/-/onetime-2.0.1.tgz

# compression
➜ npm view compression@1.7.2 dist.tarball # this is also latest
http://registry.npmjs.org/compression/-/compression-1.7.2.tgz

It looks pretty isolated to specific packages at some specific versions. Might be worthwhile to throw a follower at the registry and find any others with weird data like this.


it’s still being worked on, don’t worry. Just wait :slight_smile:

If it’s super important to make sure you have pkglocks that only do https, just use sed -i and replace them accordingly. They’re still hosted behind https.

Beyond that, I’ve no timeline or info besides the fact that it’s being tracked internally.


@zkat what’s the status here? This is a pretty significant problem … it’s a bit shocking to see so many issues around the internet for this problem and it not getting the attention it deserves.


npm install is litterally the only command we need to behave 100% with no bug, the rest: npx, npm ci, npm audit, those are new features, but npm install should our only true God

and I don’t want to switch to other package manager… we have better problem to solve than to deal with npm vs yarn :grimacing: :pray:


What I Wanted to Do

Generate a pkglock with no packages having an http resolved url

What Happened Instead

Some packages in the pkglock had http urls. Specifically, these two packages (onetime@1.1.0 and compression@1.7.2) have their dist.tarball fields as http instead of https.

Reproduction Steps

Use the package.json in this gist to generate the pkglock: https://gist.github.com/karanjthakkar/6049a743c1d3d57bb829bde87ea851a9

Details

  1. Versions:
{ npm: '5.6.0',
  ares: '1.10.1-DEV',
  cldr: '32.0',
  http_parser: '2.8.0',
  icu: '60.1',
  modules: '57',
  napi: '3',
  nghttp2: '1.32.0',
  node: '8.11.3',
  openssl: '1.0.2o',
  tz: '2017c',
  unicode: '10.0',
  uv: '1.19.1',
  v8: '6.2.414.54',
  zlib: '1.2.11' }
  1. npm info onetime@1.1.0 dist.tarball gives http://registry.npmjs.org/onetime/-/onetime-1.1.0.tgz. npm info compression@1.7.2 dist.tarball gives http://registry.npmjs.org/compression/-/compression-1.7.2.tgz.


Perfect! That’s what we have right now: a precommit hook to replace http with https. :slight_smile:


this makes me anxious

so

bump


I have updated to the latest version of NPM (6.5.0), force cleaned caches and run the install command with --prefer-online. Still, some of the URLs in package lock got converted to “http” protocol.

Am I doing something wrong?