Installing predownloaded packages offline through npm is really hard

npm has an --offline flag which will cause it to fail whenever it needs to use the internet, which I guess is in the right direction of what I want

doing npm install --offline in a project directory gives me a bunch of code ENOTCACHED failures, there are two distinct kinds:

• request to https://registry.npmjs.org/«packagename»/-/«package».tgz failed
• request to https://registry.npmjs.org/«packagename» failed

The solution for the first is fairly easy, npm allows you to install tarballs with npm install «package».tgz, or alternatively add them to the cache with npm cache add «package.tgz»

So there are no no more cache failures of the «packagename».tgz, how to deal with the metadata download issues?

How I got here

A bit more in depth, npm lets you install any packages with no dependencies fairly easily, npm intall «packagename».tgz just works! Which is reasonably expected of a package manager. An alternate but slightly more awkward workflow is npm cache add «package».tgz && npm install «packagename», which does the same thing. It gets more complicated when you include dependencies (that I believe can possibly be upgraded?).

npm will desperately try to check if any dependencies can be upgraded and fail (if you’re offline or passed the –offline flag) when it realizes the results aren’t cached. This is a bit wacky, it should just use what I have installed or something? IDK.

Solutions that I found unacceptable on the internet consisted but not limited to:

• copy your node_modules folder over to your offline machine
• Various third-party packages
• Keep your npm cache around
• Use yarn
• --cache-min=infinity/--cache-min 9999999 (these don’t seem to function any more, replaced with --offline)
• npm link would work if I had no conflicting dependencies

very related post that goes into these much more in depth

I spent a lot of time going in google circles with the “npm install offline” but somehow luckily enough I stumbled upon npm ci

The final piece: npm ci

npm ci seemed to do similarly to what npm install did but helpfully enough without the incessent “metadata files not in cache” errors. npm ci takes no arguments and just installed the project in the current directory, which means I need to add all the dependency tarballs to the cache but that should be no big deal.

Is this the end of my journey?

SHA-1 vs SHA-512

So, npm cache add hashes everything with sha512, which seems reasonable. The project I am trying to install through my package manager offline, however, includes some sha1 hashes.

It took me too long to make this connection, but npm ci kept failing on packages with sha-1 hashes. Changing that ^ line in put.js to sha1 and re-adding tarballs with npm cache add registered their sha-1 hashes and got me through the issues. Success! Almost, need to automate this better than changing npm’s source temporarily

Digging into the .npm/_cacache format, index-v5/ files look like this:

60fde9c2310b0d4cad4dab8d126b04387efba289 {"key":"pacote:tarball:file:«tarball»","integrity":sha512-«base64-gibberish hash of the tarball»","time":1604617124075,"size":«size of the tarball»)}

And a new line was added when I re-added the file with the sha-1 versions:

12b0c9014aa6f5e7cc23dd2af337ef288b0c3018 {"key":"pacote:tarball:file:«tarball»","integrity":sha1-«smaller base64-gibberish»","time":1604617124075,"size":«size of the tarball»)}

The hash at the beginning of the line is just a sha1sum of the rest of the line, from the { to the }

Oh, also, .npm/_cacache/content-v2 contains the cached tarballs addressable by their hashes:

$ sha512sum .npm/_cacache/content-v2/sha512/04/b1/692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5
04b1692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5  .npm/_cacache/content-v2/sha512/04/b1/692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5

But this is pretty reasonable to figure out

So now all I need to do is completely reimplement npm cache add for sha1. Since gentoo’s package manager is written in bash, and I’m doing this for a gentoo package, I’m re-writing this in bash.

#!/usr/bin/env bash

CACHEDIR="$(npm config get cache)/_cacache"

sha256sum() {
    command sha256sum "$@" | cut -d ' ' -f 1
}

sha1sum() {
    command sha1sum "$@" | cut -d ' ' -f 1
}

sha512sum() {
    command sha512sum "$@" | cut -d ' ' -f 1
}

hex2base64() {
    xxd -r -p | base64 -w 0
}

splithash() {
    echo "${1:0:2}/${1:2:2}/${1:4}"

}

getcachefile() {
    echo -n "$CACHEDIR/index-v5/"
    splithash "$(echo -n "pacote:tarball:file:$1" | sha256sum)"
}

newcacheline() {
    read -r -d '' JSON <<-EOF
        {"key":"pacote:tarball:file:$1","integrity":"sha1-$(sha1sum "$1" | hex2base64)","time":1604617124075,"size":$(wc -c < "$1")}
    EOF
    echo
    echo -n "$(echo -n "$JSON" | sha1sum)"
    echo -n '   '
    echo "$JSON"
}

addsha1file() {
    SHA1="$CACHEDIR/content-v2/sha1/$(splithash "$(sha1sum "$1")")"
    mkdir -p "$(dirname "$SHA1")"
    cp "$1" "$SHA1"
}

addfile() {
    addsha1file "$1"
    mkdir -p "$(dirname "$(getcachefile "$1")")"
    newcacheline "$1" >> "$(getcachefile "$1")"
}

And the end result is here. still had to patch the original to not use a git repository because that is a whole other thing.

Thanks for reading!