npm
has an --offline
flag which will cause it to fail whenever it needs to
use the internet, which I guess is in the right direction of what I want
doing npm install --offline
in a project directory gives me a bunch of code ENOTCACHED
failures, there are two distinct kinds:
request to https://registry.npmjs.org/«packagename»/-/«package».tgz failed
request to https://registry.npmjs.org/«packagename» failed
The solution for the first is fairly easy, npm allows you to install tarballs
with npm install «package».tgz
, or alternatively add them to the cache with
npm cache add «package.tgz»
So there are no no more cache failures of the «packagename».tgz
, how to deal
with the metadata download issues?
A bit more in depth, npm
lets you install any packages with no dependencies
fairly easily, npm intall «packagename».tgz
just works! Which is reasonably
expected of a package manager. An alternate but slightly more awkward workflow
is npm cache add «package».tgz && npm install «packagename»
, which does
the same thing. It gets more complicated when you include dependencies (that I
believe can possibly be upgraded?).
npm
will desperately try to check if any dependencies can be upgraded and
fail (if you're offline or passed the --offline flag) when it realizes the
results aren't cached. This is a bit wacky, it should just use what I have
installed or something? IDK.
Solutions that I found unacceptable on the internet consisted but not limited to:
node_modules
folder over to your offline machine--cache-min=infinity
/--cache-min 9999999
(these don't seem to function any more, replaced with --offline
)very related post that goes into these much more in depth
I spent a lot of time going in google circles with the "npm install offline"
but somehow luckily enough I stumbled upon npm ci
npm ci
npm ci
seemed to do similarly to what npm install
did but helpfully enough
without the incessent "metadata files not in cache" errors. npm ci
takes no
arguments and just installed the project in the current directory, which means
I need to add all the dependency tarballs to the cache but that should be no
big deal.
Is this the end of my journey?
So, npm cache add
hashes everything with
sha512
, which seems
reasonable. The project I am trying to install through my package manager
offline, however,
includes
some
sha1
hashes.
It took me too long to make this connection, but npm ci
kept failing on
packages with sha-1 hashes. Changing that
^ line in put.js
to
sha1
and re-adding tarballs with npm cache add
registered their sha-1
hashes and got me through the issues. Success! Almost, need to automate this
better than changing npm
's source temporarily
Digging into the .npm/_cacache
format, index-v5/
files look like this:
60fde9c2310b0d4cad4dab8d126b04387efba289 {"key":"pacote:tarball:file:«tarball»","integrity":sha512-«base64-gibberish hash of the tarball»","time":1604617124075,"size":«size of the tarball»)}
And a new line was added when I re-added the file with the sha-1 versions:
12b0c9014aa6f5e7cc23dd2af337ef288b0c3018 {"key":"pacote:tarball:file:«tarball»","integrity":sha1-«smaller base64-gibberish»","time":1604617124075,"size":«size of the tarball»)}
The hash at the beginning of the line is just a sha1sum
of the rest of the
line, from the {
to the }
Oh, also, .npm/_cacache/content-v2
contains the cached tarballs addressable
by their hashes:
$ sha512sum .npm/_cacache/content-v2/sha512/04/b1/692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5
04b1692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5 .npm/_cacache/content-v2/sha512/04/b1/692c170df913ca52c171a9190c8b0a9338e762f93acaaf93a92c9f6a0a751d4b14ba4bcd3e4115ff98885f47aeaed1891ce104e445e69e8ce33620b87ef5
But this is pretty reasonable to figure out
So now all I need to do is completely reimplement npm cache add
for sha1.
Since gentoo's package manager is written in bash, and I'm doing this for a
gentoo package, I'm re-writing this in bash.
#!/usr/bin/env bash
CACHEDIR="$(npm config get cache)/_cacache"
sha256sum() {
command sha256sum "$@" | cut -d ' ' -f 1
}
sha1sum() {
command sha1sum "$@" | cut -d ' ' -f 1
}
sha512sum() {
command sha512sum "$@" | cut -d ' ' -f 1
}
hex2base64() {
xxd -r -p | base64 -w 0
}
splithash() {
echo "${1:0:2}/${1:2:2}/${1:4}"
}
getcachefile() {
echo -n "$CACHEDIR/index-v5/"
splithash "$(echo -n "pacote:tarball:file:$1" | sha256sum)"
}
newcacheline() {
read -r -d '' JSON <<-EOF
{"key":"pacote:tarball:file:$1","integrity":"sha1-$(sha1sum "$1" | hex2base64)","time":1604617124075,"size":$(wc -c < "$1")}
EOF
echo
echo -n "$(echo -n "$JSON" | sha1sum)"
echo -n ' '
echo "$JSON"
}
addsha1file() {
SHA1="$CACHEDIR/content-v2/sha1/$(splithash "$(sha1sum "$1")")"
mkdir -p "$(dirname "$SHA1")"
cp "$1" "$SHA1"
}
addfile() {
addsha1file "$1"
mkdir -p "$(dirname "$(getcachefile "$1")")"
newcacheline "$1" >> "$(getcachefile "$1")"
}
And the end result is here. still had to patch the original to not use a git repository because that is a whole other thing.
Thanks for reading!