My failed attempt to shrink all NPM packages by 5%

The amount of modules my docker hosts download from npm is anything but tiny.

I can speak to this - there is no meaningful decompression effect across an insane set of tested data at Google and elsewhere. Zopfli was invented prior to brotli

Zopfli is easiest to think of as something that just tries harder than gzip to find matches and better encodings. Much harder.

decompression speed is linear either way.

It's easiest to think of decompression as a linear time vm executor[1], where the bytecoded instructions are basically

go back bytes, output the next bytes you see, then output character

(outputting literal data is the instruction )

Assuming you did not output a file larger than the original uncompressed file (why would you bother?), you will, worst case, process N bytes during decompression, where N is the size of the original input file.

The practical decompression speed is driven by cache behavior, but it thrashes the cache no matter what.

In practice, reduction of size vs gzip occurs by either finding larger runs, or encodings that are smaller than the existing ones.

After all, if you want the compressed file to shrink, you need output less instructions somehow, or make more of the instructions identical (so they can be represented in less bits by later huffman coding).

In practice, this has almost exclusively positive effects on decompression speed - either the vm has less things to process (which is faster), or more of the things it does look the same (which has better cache behavior).

[1] this is one way archive formats will sometimes choose to deal with multiple compression method support - encode them all to the same kind of bytecode (usually some form of copy + literal instruction set), and then decoding is the same for all of them. ~all compression algorithms output some bytecode like the above on their own already, so it's not a lot of work. This doesn't help you support other archive formats, but if you want to have a bunch of per-file compression options that you pick from based on what works best, this enables you to still only have to have one decoder.

zopfli is the wrong thing to use here.

If you want an example of where these things go badly:

the standard compression level for rpms on redhat distros is zstd level 19.

This has all the downsides of other algorithms - it's super slow - often 75-100x slower than the default level of zstd[1]. It achieves a few percent more compression for that speed. compared to even level 10, it's like 0-1% higher compression, but 20x slower.

This is bad enough - the kicker is that at this level, it's slower than xz for ~all cases, and xz is 10% smaller. The only reason to use zstd is because you want fairly good compression, but fast.

So here, they've chosen to use it in a way that compresses really slowly, but gives you none of the benefit of compressing really slowly.

Now, unlike the npm case, there was no good reason to choose level 19 - there were no backwards compatibility constraints driving it, etc. I went through the PR history on this change, it was not particularly illuminating (it does not seem lots of thought was given to the level choice).

I mention all this because it has a real effect on the experience of building rpms - this is why it takes eons to make kernel debuginfo rpms on fedora. Or any large RPM. Almost all time is spent compressing it with zstd at level 19. On my computer this takes many minutes. If you switch it to use even xz, it will do it about 15-20x faster (single threaded. if you thread both of them, xz will win by even more, because of how slow the setting is for zstd. If you use reasonable settings for zstd, obviously, it achieves gigabytes/second in parallel mode)

Using zopfli would be like choosing level 19 zstd for npm. While backwards compatibility is certainly painful to deal with here, zopfli is not likely better than doing nothing. You will make certain cases just insanely slow. You will save someone's bandwidth, but in exchange you will burn insane amounts of developer CPU.

zopfli is worse than level 19, it can often be 100x-200x slower than gzip -9.

Doesn't npm support insanely relaxed/etc scripting hooks anyway?

If so, if backwards compatibility is your main constraint, you would be "better off" double compressing (IE embed xz or whatever + a bootstrap decompressor and using the hooks to decompress it on old versions of npm). Or just shipping .tar.gz's than, when run, fetch the .xz and decompress it on older npm.

Or you know, fish in the right pond - you would almost certainly achieve much higher reductions by enforcing cleaner shipping packages (IE not including random garbage, etc) than by compressing the garbage more.

[1] on my computer, single threaded level 19 does 7meg/second, the default does 500meg/second. Level 10 does about 130meg/second.

If OP wanted to shrink nom packages then npm could introduce two types of npm package - the build package and source one. This way a lot of packages would be smaller, because package code could be safely distributed through a separate package and not kept in the build package. There are a lot npm packages that explicitly include whole git repo in the package and they do so because there's only one type of package they can use