installing with git:// URL results in spurious .git directory within node_modules

What I Wanted to Do

After creating a fresh project:

mkdir -p ~/temp/foo
cd ~/temp/foo
npm init -y

I want and expect either one of the following to produce the exact same result:

npm i websocket@git://github.com/frozeman/WebSocket-Node.git#browserifyCompatible
npm i websocket@github:frozeman/WebSocket-Node#browserifyCompatible

[ Don’t run both of the above, just one or the other. ]

However, after running install with the git:// URL then there is a spurious .git/ directory inside node_modules/websocket/. And in that case additional package installs will fail:

$ npm i rxjs
npm ERR! path /Users/michael/temp/foo/node_modules/websocket
npm ERR! code EISGIT
npm ERR! git /Users/michael/temp/foo/node_modules/websocket: Appears to be a git repo or submodule.
npm ERR! git     /Users/michael/temp/foo/node_modules/websocket
npm ERR! git Refusing to remove it. Update manually,
npm ERR! git or move it out of the way first.

npm ERR! A complete log of this run can be found in:
npm ERR!     /Users/michael/.npm/_logs/2019-07-12T18_57_17_101Z-debug.log

There is no such problem if the first install was done with the github: specifier since afterward there is no .git/ directory inside node_modules/websocket.

The same problem occurs if websocket is listed as a dependency in package.json with the git:// URL, and like the cli scenario the problem does not occur if the github: specifier is used instead.

What Happened Instead

Described above.

Reproduction Steps

Given above.

Details

I have tried with several combinations of node and npm versions and always experience the same problem as described above when the git:// URL is used instead of the github: specifier.

Platform Info

$ npm --versions
{ npm: '6.10.1',
  ares: '1.15.0',
  brotli: '1.0.7',
  cldr: '35.1',
  http_parser: '2.8.0',
  icu: '64.2',
  modules: '64',
  napi: '4',
  nghttp2: '1.34.0',
  node: '10.16.0',
  openssl: '1.1.1b',
  tz: '2019a',
  unicode: '12.1',
  uv: '1.28.0',
  v8: '6.8.275.32-node.52',
  zlib: '1.2.11' }

$ node -p process.platform
darwin

I was unable to reproduce this using the instructions with same version of npm and node, and also on Mac.

Speculation. There were (are?) some issues with .git folder getting incorrectly included because it has a branch name or filename that incorrectly cause the blacklist to get overridden. When you get the bad install, is it a functional .git repo or does it have just a few files/folders?

To clarify, there were no problems when you used the git:// URL?

After installing from the git:// URL did you ls -d node_modules/websocket/.*?
Did you see node_modules/websocket/.git in the output of ls?

I thought maybe there was some weird cache behavior that allowed you to install with the git:// URL cleanly, assuming maybe you had never before installed that package using the URL.

So I tried with docker:

$ docker run -it --rm node:10.16.0 bash
root@32e7c21f5a98:/# node --version
v10.16.0
root@32e7c21f5a98:/# npm --version
6.9.0
root@32e7c21f5a98:/# mkdir -p temp/foo
root@32e7c21f5a98:/# cd temp/foo
root@32e7c21f5a98:/temp/foo# npm init -y
Wrote to /temp/foo/package.json:

{
  "name": "foo",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC"
}


root@32e7c21f5a98:/temp/foo# npm i websocket@git://github.com/frozeman/WebSocket-Node.git#browserifyCompatible

> websocket@1.0.26 install /temp/foo/node_modules/websocket
> (node-gyp rebuild 2> builderror.log) || (exit 0)

make: Entering directory '/temp/foo/node_modules/websocket/build'
  CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
  SOLINK_MODULE(target) Release/obj.target/bufferutil.node
  COPY Release/bufferutil.node
  CXX(target) Release/obj.target/validation/src/validation.o
  SOLINK_MODULE(target) Release/obj.target/validation.node
  COPY Release/validation.node
make: Leaving directory '/temp/foo/node_modules/websocket/build'
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN foo@1.0.0 No description
npm WARN foo@1.0.0 No repository field.

+ websocket@1.0.26
added 7 packages from 15 contributors and audited 7 packages in 3.839s
found 0 vulnerabilities

root@32e7c21f5a98:/temp/foo# ls -ld node_modules/websocket/.*
drwxr-xr-x 7 root root 4096 Jul 13 00:19 node_modules/websocket/.
drwxr-xr-x 9 root root 4096 Jul 13 00:19 node_modules/websocket/..
drwxr-xr-x 2 root root 4096 Jul 13 00:19 node_modules/websocket/.git
-rw-r--r-- 1 root root 5821 Oct 26  1985 node_modules/websocket/.jshintrc
root@32e7c21f5a98:/temp/foo# npm i rxjs
npm ERR! path /temp/foo/node_modules/websocket
npm ERR! code EISGIT
npm ERR! git /temp/foo/node_modules/websocket: Appears to be a git repo or submodule.
npm ERR! git     /temp/foo/node_modules/websocket
npm ERR! git Refusing to remove it. Update manually,
npm ERR! git or move it out of the way first.

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2019-07-13T00_20_08_198Z-debug.log

Note this clean environment had the same problem I reported.

I was a bit surprised too because your reproduce steps looked straight forward. And you have reproduced in docker too.

$ npm --version
6.10.1

$ npm cache clear --force
...
$ npm init -y
...
$ npm i websocket@git://github.com/frozeman/WebSocket-Node.git#browserifyCompatible

> websocket@1.0.26 install /Users/john/Documents/Sandpits/npm.community/8815/node_modules/websocket
> (node-gyp rebuild 2> builderror.log) || (exit 0)

  CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
  SOLINK_MODULE(target) Release/bufferutil.node
  CXX(target) Release/obj.target/validation/src/validation.o
  SOLINK_MODULE(target) Release/validation.node
npm notice created a lockfile as package-lock.json. You should commit this file.
npm WARN 8815@1.0.0 No description
npm WARN 8815@1.0.0 No repository field.

+ websocket@1.0.26
added 7 packages from 15 contributors and audited 7 packages in 5.937s
found 0 vulnerabilities

$ ls -d node_modules/websocket/.*
node_modules/websocket/.		node_modules/websocket/..		node_modules/websocket/.jshintrc

$ npm i rxjs
npm WARN 8815@1.0.0 No description
npm WARN 8815@1.0.0 No repository field.

+ rxjs@6.5.2
added 2 packages from 7 contributors and audited 9 packages in 1.466s
found 0 vulnerabilities

I was able to reproduce your steps using docker (with npm 6.9.0).

The only file in the .git directory is index, and the “main” entry point in your package.json is also index. I think I saw a report like that before…

Nice catch!! I’ll leave a comment/s on that GH issue.

1 Like

See: https://github.com/npm/npm-packlist/issues/35#issuecomment-511072832

The key thing is that npm cache clear --force is necessary in the case where a package was previously installed, with an older version of npm, from a git:// URL and had this problem.