Thu, 5 Nov 2020 20:03:38 -0800
npm
has an --offline
flag which will cause it to fail whenever it needs to use the internet, which I guess is in the right direction of what I want
doing npm install --offline
in a project directory gives me a bunch of code ENOTCACHED
failures, there are two distinct kinds:
request to https://registry.npmjs.org/«packagename»/-/«package».tgz failed
request to https://registry.npmjs.org/«packagename» failed
The solution for the first is fairly easy, npm allows you to install tarballs with npm install «package».tgz
, or alternatively add them to the cache with npm cache add «package.tgz»
So there are no no more cache failures of the «packagename».tgz
, how to deal with the metadata download issues?
A bit more in depth, npm
lets you install any packages with no dependencies fairly easily, npm intall «packagename».tgz
just works! Which is reasonably expected of a package manager. An alternate but slightly more awkward workflow is npm cache add «package».tgz && npm install «packagename»
, which does the same thing. It gets more complicated when you include dependencies (that I believe can possibly be upgraded?).
npm
will desperately try to check if any dependencies can be upgraded and fail (if you’re offline or passed the –offline flag) when it realizes the results aren’t cached. This is a bit wacky, it should just use what I have installed or something? IDK.
Solutions that I found unacceptable on the internet consisted but not limited to:
node_modules
folder over to your offline machine--cache-min=infinity
/--cache-min 9999999
(these don’t seem to function any more, replaced with --offline
)very related post that goes into these much more in depth
I spent a lot of time going in google circles with the “npm install offline” but somehow luckily enough I stumbled upon npm ci
npm ci
npm ci
seemed to do similarly to what npm install
did but helpfully enough without the incessent “metadata files not in cache” errors. npm ci
takes no arguments and just installed the project in the current directory, which means I need to add all the dependency tarballs to the cache but that should be no big deal.
Is this the end of my journey?
So, npm cache add
hashes everything with sha512
, which seems reasonable. The project I am trying to install through my package manager offline, however, includes some sha1 hashes.
It took me too long to make this connection, but npm ci
kept failing on packages with sha-1 hashes. Changing that ^ line in put.js
to sha1
and re-adding tarballs with npm cache add
registered their sha-1
hashes and got me through the issues. Success! Almost, need to automate this better than changing npm
’s source temporarily
Digging into the .npm/_cacache
format, index-v5/
files look like this:
60fde9c2310b0d4cad4dab8d126b04387efba289 {"key":"pacote:tarball:file:«tarball»","integrity":sha512-«base64-gibberish hash of the tarball»","time":1604617124075,"size":«size of the tarball»)}
And a new line was added when I re-added the file with the sha-1 versions:
12b0c9014aa6f5e7cc23dd2af337ef288b0c3018 {"key":"pacote:tarball:file:«tarball»","integrity":sha1-«smaller base64-gibberish»","time":1604617124075,"size":«size of the tarball»)}
The hash at the beginning of the line is just a sha1sum
of the rest of the line, from the {
to the }
Oh, also, .npm/_cacache/content-v2
contains the cached tarballs addressable by their hashes:
sha512sum .npm/_cacache/content-v2/sha512/04/b1/692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5
$ 04b1692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5 .npm/_cacache/content-v2/sha512/04/b1/692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5
But this is pretty reasonable to figure out
So now all I need to do is completely reimplement npm cache add
for sha1. Since gentoo’s package manager is written in bash, and I’m doing this for a gentoo package, I’m re-writing this in bash.
Some bash code you might not want to see
This actually isn’t too bad and is twice as fast as npm cache add
which I’m happy about:
#!/usr/bin/env bash
CACHEDIR="$(npm config get cache)/_cacache"
sha256sum() {
command sha256sum "$@" | cut -d ' ' -f 1
}
sha1sum() {
command sha1sum "$@" | cut -d ' ' -f 1
}
sha512sum() {
command sha512sum "$@" | cut -d ' ' -f 1
}
hex2base64() {
xxd -r -p | base64 -w 0
}
splithash() {
echo "${1:0:2}/${1:2:2}/${1:4}"
}
getcachefile() {
echo -n "$CACHEDIR/index-v5/"
splithash "$(echo -n "pacote:tarball:file:$1" | sha256sum)"
}
newcacheline() {
read -r -d '' JSON <<-EOF
$1","integrity":"sha1-$(sha1sum "$1" | hex2base64)","time":1604617124075,"size":$(wc -c < "$1")}
{"key":"pacote:tarball:file:
EOF
echo$(echo -n "$JSON" | sha1sum)"
echo -n "
echo -n ' '$JSON"
echo "
}
addsha1file() {$CACHEDIR/content-v2/sha1/$(splithash "$(sha1sum "$1")")"
SHA1="mkdir -p "$(dirname "$SHA1")"
$1" "$SHA1"
cp "
}
addfile() {$1"
addsha1file "$(dirname "$(getcachefile "$1")")"
mkdir -p "$1" >> "$(getcachefile "$1")"
newcacheline " }
And the end result is here. still had to patch the original to not use a git repository because that is a whole other thing.
Thanks for reading!