Packages published on UNIX with `:` in filenames fail to be required on Windows

cli
priority:low
triaged

(Benjamin Lupton) #1

Published editions v2.0.1 and getmac v1.4.5 via travisci. They create directories that have colons within them, such as edition:esnext and edition:node:0.8. This works fine on unix systems like travisci builds, as well as macos which is what I develop on. However, it turns out Windows users end up getting Cannot find module errors.

Booting up a windows VM, it turns out the packages install fine. It is just the the require calls fail to resolve anything. It also seems that npm plays some trickery with the filenames, as instead of using colons (which is an invalid character on windows filesystems) they now use an UTF character, as seen from the screenshot:

I would have thought that if npm is going to do some trickery with the filenames, that it would do so with a trick that still works with the node module resolution system. Thinking of how it should work would be:

  1. Invalid characters are unpacked by npm on windows as underscores
  2. Invalid characters within the node.js module resolver are swapped to underscores
  3. Everything works

If that is infeasible, then it would be nice if there was a harder failure somewhere, so that this error could have been caught way sooner and identified way sooner. For instance, it would have been nice if npm just straight out fails to install/unpack such packages (with a clear error message as to why), rather than unpacking them with filename trickery which made debugging harder. Or some type of warning somewhere, before publish, that the package wouldn’t/may not work on windows, and whether or not we want to continue anyway.

Platform versions

My macOS development environment:

> npm --versions
{ npm: '6.3.0',
  ares: '1.14.0',
  cldr: '33.1',
  http_parser: '2.8.0',
  icu: '62.1',
  modules: '64',
  napi: '3',
  nghttp2: '1.32.0',
  node: '10.8.0',
  openssl: '1.1.0h',
  tz: '2018e',
  unicode: '11.0',
  uv: '1.22.0',
  v8: '6.7.288.49-node.19',
  zlib: '1.2.11' }

> uname -a
Darwin balbook-2018.local 17.7.0 Darwin Kernel Version 17.7.0: Thu Jun 21 22:53:14 PDT 2018; root:xnu-4570.71.2~1/RELEASE_X86_64 x86_64

Travis details that published them

Windows system that is trying to consume them

{ editions: '2.0.1',
  npm: '6.2.0',
  ares: '1.14.0',
  cldr: '33.1',
  http_parser: '2.8.0',
  icu: '62.1',
  modules: '64',
  napi: '3',
  nghttp2: '1.32.0',
  node: '10.9.0',
  openssl: '1.1.0i',
  tz: '2018e',
  unicode: '11.0',
  uv: '1.22.0',
  v8: '6.8.275.24-node.14',
  zlib: '1.2.11' }

Fresh windows 10 1803 64 bit vm


build dir name influences build success
(Rebecca Turner) #2

It’s not npm doing the trickery here. I’m not actually sure what layer is doing the translation, but I suspect it’s the Windows system-call level. (Node tends to not do trickery like this either.)


(Kat Marchán) #3

It’s node-tar doing the translation.


(Kat Marchán) #4

Also note: : is a bad idea even on Unix. It’s going to fuck with $PATH if you ever put an executable behind one of these paths, and there’s literally no way to escape that at all. There’s other cases of filesystem stuff that you really shouldn’t do but we’re unlikely (or at least reticent) to enforce: variant-case filenames, unicode characters, etc. I don’t even know all of 'em but it will bite folks sometimes.


(Rebecca Turner) #5

Wow that node-tar thing is weird and clever and I’m not sure it’s really desirable.


(Benjamin Lupton) #6

Agreed. I think having node-tar provide a flag which will throw an error if a rename is required, that then npm can use by default, makes the most sense. As a hard fail is more desirable (or at least a warning, or confirmation) for npm packages.


(Benjamin Lupton) #7

Until there is a better option, I’ve published valid-directory, which facilitates a preventative measure:

  1. npm install --save-dev valid-directory
  2. Add && npx valid-directory to your test script in package.json

(system) #8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.