Thu, 5 Nov 2020 20:03:38 -0800
npm
has an --offline
flag which will cause
it to fail whenever it needs to use the internet, which I guess is in
the right direction of what I want
doing npm install --offline
in a project directory gives
me a bunch of code ENOTCACHED
failures, there are two
distinct kinds:
request to https://registry.npmjs.org/«packagename»/-/«package».tgz failed
request to https://registry.npmjs.org/«packagename» failed
The solution for the first is fairly easy, npm allows you to install
tarballs with npm install «package».tgz
, or alternatively
add them to the cache with npm cache add «package.tgz»
So there are no no more cache failures of the
«packagename».tgz
, how to deal with the metadata download
issues?
A bit more in depth, npm
lets you install any packages
with no dependencies fairly easily,
npm intall «packagename».tgz
just works! Which is
reasonably expected of a package manager. An alternate but slightly more
awkward workflow is
npm cache add «package».tgz && npm install «packagename»
,
which does the same thing. It gets more complicated when you include
dependencies (that I believe can possibly be upgraded?).
npm
will desperately try to check if any dependencies
can be upgraded and fail (if you’re offline or passed the –offline flag)
when it realizes the results aren’t cached. This is a bit wacky, it
should just use what I have installed or something? IDK.
Solutions that I found unacceptable on the internet consisted but not limited to:
node_modules
folder over to your offline machine--cache-min=infinity
/--cache-min 9999999
(these don’t seem to function any more, replaced with
--offline
)very related post that goes into these much more in depth
I spent a lot of time going in google circles with the “npm install
offline” but somehow luckily enough I stumbled upon
npm ci
npm ci
npm ci
seemed to do similarly to what
npm install
did but helpfully enough without the incessent
“metadata files not in cache” errors. npm ci
takes no
arguments and just installed the project in the current directory, which
means I need to add all the dependency tarballs to the cache but that
should be no big deal.
Is this the end of my journey?
So, npm cache add
hashes
everything with sha512
, which seems reasonable. The
project I am trying to install through my package manager offline,
however, includes
some
sha1
hashes.
It took me too long to make this connection, but npm ci
kept failing on packages with sha-1 hashes. Changing that ^
line in put.js
to sha1
and re-adding tarballs
with npm cache add
registered their sha-1
hashes and got me through the issues. Success! Almost, need to automate
this better than changing npm
’s source temporarily
Digging into the .npm/_cacache
format,
index-v5/
files look like this:
60fde9c2310b0d4cad4dab8d126b04387efba289 {"key":"pacote:tarball:file:«tarball»","integrity":sha512-«base64-gibberish hash of the tarball»","time":1604617124075,"size":«size of the tarball»)}
And a new line was added when I re-added the file with the sha-1 versions:
12b0c9014aa6f5e7cc23dd2af337ef288b0c3018 {"key":"pacote:tarball:file:«tarball»","integrity":sha1-«smaller base64-gibberish»","time":1604617124075,"size":«size of the tarball»)}
The hash at the beginning of the line is just a sha1sum
of the rest of the line, from the {
to the
}
Oh, also, .npm/_cacache/content-v2
contains the cached
tarballs addressable by their hashes:
$ sha512sum .npm/_cacache/content-v2/sha512/04/b1/692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5
04b1692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5 .npm/_cacache/content-v2/sha512/04/b1/692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5
But this is pretty reasonable to figure out
So now all I need to do is completely reimplement
npm cache add
for sha1. Since gentoo’s package manager is
written in bash, and I’m doing this for a gentoo package, I’m re-writing
this in bash.
This actually isn’t too bad and is twice as fast as
npm cache add
which I’m happy about:
#!/usr/bin/env bash
CACHEDIR="$(npm config get cache)/_cacache"
sha256sum() {
command sha256sum "$@" | cut -d ' ' -f 1
}
sha1sum() {
command sha1sum "$@" | cut -d ' ' -f 1
}
sha512sum() {
command sha512sum "$@" | cut -d ' ' -f 1
}
hex2base64() {
xxd -r -p | base64 -w 0
}
splithash() {
echo "${1:0:2}/${1:2:2}/${1:4}"
}
getcachefile() {
echo -n "$CACHEDIR/index-v5/"
splithash "$(echo -n "pacote:tarball:file:$1" | sha256sum)"
}
newcacheline() {
read -r -d '' JSON <<-EOF
{"key":"pacote:tarball:file:$1","integrity":"sha1-$(sha1sum "$1" | hex2base64)","time":1604617124075,"size":$(wc -c < "$1")}
EOF
echo
echo -n "$(echo -n "$JSON" | sha1sum)"
echo -n ' '
echo "$JSON"
}
addsha1file() {
SHA1="$CACHEDIR/content-v2/sha1/$(splithash "$(sha1sum "$1")")"
mkdir -p "$(dirname "$SHA1")"
cp "$1" "$SHA1"
}
addfile() {
addsha1file "$1"
mkdir -p "$(dirname "$(getcachefile "$1")")"
newcacheline "$1" >> "$(getcachefile "$1")"
}
And the end result is here. still had to patch the original to not use a git repository because that is a whole other thing.
Thanks for reading!