I wonder if you could get better results if you built a dictionary over entire npm. I suspect most common words could easily be reduced to 16k word index. Would be much faster, dictionary would probably fit in cache, can even optimize it in memory for cache prefetch.
My failed attempt to shrink all NPM packages by 5%
161–170 of 253 posts
Re: My failed attempt to shrink all NPM packages by 5%
#162Pulling on this thread, there are a few people who have looked at the ways zopfli is inefficient. Including this guy who forked it, and tried to contribute a couple improvements back to master:
https://github.com/fhanau/Efficient-Compression-Tool
These days if you’re going to iterate on a solution you’d better make it multithreaded. We have laptops where sequential code uses 8% of the available cpu.
Re: My failed attempt to shrink all NPM packages by 5%
#163Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.
It's not even funny: $ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/* /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux: .r-xr-xr-x 129k root 1 Jan 1970 xsel /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows: .r-xr-xr-x 444k root 1 Jan 1970 clipboard_i686.exe .r-xr-xr-x 331k root 1 Jan 1970 clip…
Are they reproducible? Shipping binaries in JS packages is dodgy AF - a Jia Tan attack waiting to happen.
Re: My failed attempt to shrink all NPM packages by 5%
#164Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.
One of the things I like about node_modules is that it's not purely source code and it's not purely build artifacts.
You can read the code and you can usually read the actual README/docs/tests of the package instead of having to find it online. And you can usually edit library code for debugging purposes.
If node_modules is taking up a lot of space across a bunch of old projects, just write the `find` script that recursively deletes them all; You can always run `npm install` in the future when you need to work on that project again.
Re: My failed attempt to shrink all NPM packages by 5%
#165Years back I came to the conclusion that conda using bzip2 for compression was a big mistake. Back then if you wanted to use a particular neural network it was meant for a certain version of Tensorflow which expected you to have a certain version of the CUDA libs. If you had to work with multiple models the "normal" way to do things was use the developer unfriendly [1][2] installers from NVIDIA to install a single ve…
>But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time.
For Paper, I'm planning to cache both the wheel archives (so that they're available without recompressing on demand) and unpacked versions (installing into new environments will generally use hard links to the unpacked cache, where possible).
> If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.
FWIW, in my testing LZMA is a big win (and I'm sure zstd would be as well, but LZMA has standard library support already). But there are serious roadblocks to adopting a change like that in the Python ecosystem. This sort of idea puts them several layers deep in meta-discussion - see for example https://discuss.python.org/t/pep-777-how-to-re-invent-the-wh... . In general, progress on Python packaging gets stuck in a double-bind: try to change too little and you won't get any buy-in that it's worthwhile, but try to change too much and everyone will freak out about backwards compatibility.
Re: My failed attempt to shrink all NPM packages by 5%
#166I wonder if you could get better results if you built a dictionary over entire npm. I suspect most common words could easily be reduced to 16k word index. Would be much faster, dictionary would probably fit in cache, can even optimize it in memory for cache prefetch.
This seems like a non-starter to me - new packages are added to npm all the time, and will alter the word frequency distribution. If you aren't prepared to re-build constantly and accept that the dictionary isn't optimal, then it's hard to imagine it being significantly better than what you build with a more naive approach. Basically - why try to fine-tune to a moving target?
Re: My failed attempt to shrink all NPM packages by 5%
#167Earlier quoted context omitted.
It's not even funny: $ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/* /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux: .r-xr-xr-x 129k root 1 Jan 1970 xsel /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows: .r-xr-xr-x 444k root 1 Jan 1970 clipboard_i686.exe .r-xr-xr-x 331k root 1 Jan 1970 clip…
Are they reproducible? Shipping binaries in JS packages is dodgy AF - a Jia Tan attack waiting to happen.
The executables are vendored in the repo [0].
[0] https://github.com/sindresorhus/clipboardy/tree/main/fallbac...
Re: My failed attempt to shrink all NPM packages by 5%
#168Does npm even default to gzip -9? Wikipedia claims zopfli is 80 times slower under default settings.
My experience has been that past -6 or so, gzip files get only a tiny bit smaller in typical cases. (I think I've even seen them get bigger with -9.)
Re: My failed attempt to shrink all NPM packages by 5%
#169This reminds me of a time I lost an argument with John-David Dalton about cleaning up/minifying lodash as an npm dependency, because when including the readme and license for every sub-library, a lodash import came to ~2.5MB at the time. This also took a lot of seeking time for disks because there were so many individual files.
The conversation started and ended at the word cache.
Re: My failed attempt to shrink all NPM packages by 5%
#170I'd love to see an effort like like this succeed in the Python ecosystem. Right now, PyPI is dependent upon Fastly to serve files, on the order of >1 petabyte per day. That's a truly massive in-kind donation, compared to the PSF's operating budget (only a few million dollars per year - far smaller than Linux or Mozilla).