Integrity check failing for local tarball packages on CI server

(Joe Attardi) #1

I have a mono-repo that contains 3 projects: the main project, and 2 dependent libraries.

We are currently trying to switch from yarn to npm (yay!), but I’m running into some problems.

With yarn, we just referenced the dependent libraries by their relative paths, and yarn made a copy of these directories. With npm, it seems to make a symlink instead - which, for reasons still unknown, makes our Angular app fail to build.

The best solution I could think of was to use npm pack. So when building the dependent projects, we run npm pack to create a tarball. Then, in the main project’s package.json, we reference the filenames of these tarballs. Locally, it all works as expected. It pulls in and extracts the tarballs when running npm install.

The problem is when we try to run npm install on the main project on our Drone CI server. The dependent projects are built fine, but when we run npm install on the main project, it fails saying “Integrity check failed”. It then shows the sha512 that was calculated vs the sha512 that was stored in package-lock.json.

I’m assuming that the reason for this is that the gzipped tar file contains the timestamp in the header, so every time the tarball is created it results in a different sha512.

I have two questions:

  1. Why don’t I get the integrity check failed error when I run npm install locally?
  2. Is there anything I can to do work around this issue?

Any ideas would be appreciated - thank you!

(Joe Attardi) #2

Sorry, I missed some details that would probably help:

  • npm version 5.6.0
  • local machine is a MacBook Pro
  • CI server is a docker container running Alpine Linux

(Lars Willighagen) #3

I don’t think timestamps are actually used in the version of gzip that npm uses – packing the same package on the same machine twice, with several minutes in between, produced the same integrity and hash. However, the OS does matter (proof), as someone pointed out to me a while ago after I chased ghosts bugs for way to long.

(Joe Attardi) #4

Funny you should reply here! While googling this problem I found your post, and I bet that’s the problem since the platforms are different. Did you ever find a solution?

(Lars Willighagen) #5

No, I haven’t, and didn’t have the need to, either. The actual gzip handling is layered a few dependencies down, so changing the behaviour with npm would be quite something too. I also don’t know if the OS code is actually required. Anyway, there might be a workaround, but I think you’d be better off with a different solution to your original problem.

First of all, maybe npm v6.4.1 magically works, but I don’t know if that’s possible in your situation. If not, or if it doesn’t work, I can try to reproduce the Angular problem, and try to figure out what the cause is, and if that is avoidable. I’d probably need some extra info on the build though, as I’m not too familiar with Angular.

(Joe Attardi) #6

I am in the midst of trying another solution. I noticed that you can also specify the path to a .tar file. Tar files don’t have a flag for the OS (though they do have ones for the user and group).

I wrote a custom tar utility using the tar npm package, which supports the notion of a “portable” tar file, one which leaves out the machine specific metadata.

As before, it works locally on my machine. But I still get the integrity check failure on the CI server. Still trying to figure out why it would get a different checksum.

Though one thing I noticed - in package-lock.json, I checked the checksum for one of my tar files. It is a string that starts with sha512-blahblahblah so I assumed it was just a SHA512 sum.

But when I ran shasum -a 512 myfile.tar, it gives me a different checksum than the one in the package-lock.json.

Is it not just a simple SHA512 sum that npm uses for the integrity check?

(Joe Attardi) #7

Ah, I think I’ve figured it out.

When I create the tar file, even specifying the portable option and setting the mtime to 0, the individually added files still have their timestamps.

Since the tarball is created from the dist directory which is the output of the build process, whenever the build is run the files will have new timestamps, which I imagine results in a different tar file which would result in a different hash.

Will have to keep digging for a solution.

(system) #8

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.