I’m more concerned about the 40gb of node_modules. Why hasn’t node supported tgz node_modules. That would save 75% of the space or more.
isn't this a file system thing? Why bake it into npm?
efficiency
My failed attempt to shrink all NPM packages by 5%
181–190 of 253 posts
Re: My failed attempt to shrink all NPM packages by 5%
#182Earlier quoted context omitted.
I think "massive" is overstating it. I don't think deploying a new version of a package is something that happens many times a day, so it wouldn't be a constant pain point. Also, since this is a case of having something compressed once and decompressed potentially thousands of times, it seems like the perfect tool for the job.
Every build in a CI system would probably create the package. This is changing every build in every CI system to make it slower.
Just use it on the release build.
Re: My failed attempt to shrink all NPM packages by 5%
#183I created once a maven plugin to recompress Java artefacts with zopfli. I rewrote it in Java and runs entirely in the JVM. This means, the speed is worse and may contain bugs:
Re: My failed attempt to shrink all NPM packages by 5%
#184Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.
It's not even funny: $ ll /nix/store/*-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/* /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/linux: .r-xr-xr-x 129k root 1 Jan 1970 xsel /nix/store/…-insect-5.9.0/lib/node_modules/insect/node_modules/clipboardy/fallbacks/windows: .r-xr-xr-x 444k root 1 Jan 1970 clipboard_i686.exe .r-xr-xr-x 331k root 1 Jan 1970 clip…
I don't know why, but clipboard libraries tend to be really poorly implemented, especially in scripting languages.
I just checked out clipboardy and all they do is dispatch binaries from the path and hope it's the right one (or if it's even there at all). I think I had a similar experience with Python and Lua scripts. There's an unfunny amount of poorly-written one-off clipboard scripts out there just waiting to be exploited.
I'm only glad that the go-to clipboard library in Rust (arboard) seems solid.
Re: My failed attempt to shrink all NPM packages by 5%
#185I'd love to see an effort like like this succeed in the Python ecosystem. Right now, PyPI is dependent upon Fastly to serve files, on the order of >1 petabyte per day. That's a truly massive in-kind donation, compared to the PSF's operating budget (only a few million dollars per year - far smaller than Linux or Mozilla).
No problem, I'm sure if Fastly stopped doing it JiaTanCo would step up
Re: My failed attempt to shrink all NPM packages by 5%
#186I wonder if you could get better results if you built a dictionary over entire npm. I suspect most common words could easily be reduced to 16k word index. Would be much faster, dictionary would probably fit in cache, can even optimize it in memory for cache prefetch.
This seems like a non-starter to me - new packages are added to npm all the time, and will alter the word frequency distribution. If you aren't prepared to re-build constantly and accept that the dictionary isn't optimal, then it's hard to imagine it being significantly better than what you build with a more naive approach. Basically - why try to fine-tune to a moving target?
But is it really moving that fast ? I suspect most fundamental terms in programming and the variations do not change often. You will always have keywords, built ins and the most popular concepts from libs/frameworks.
So it is basically downloading a few hundred kb dictionary every year ?
Re: My failed attempt to shrink all NPM packages by 5%
#187What about a different approach - an optional npm proxy that recompresses popular packages with 7z/etc in the background? Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready. Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packa…
This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller th…
Enjoy zipbomb.js
Re: My failed attempt to shrink all NPM packages by 5%
#188Re: My failed attempt to shrink all NPM packages by 5%
#189Earlier quoted context omitted.
This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller th…
Enjoy zipbomb.js
But npm already decompresses every package because it shows the contents on its website. So yeah it can be malicious but it already has dealt with that risk.
Re: My failed attempt to shrink all NPM packages by 5%
#190Earlier quoted context omitted.
https://www.reddit.com/r/webdev/comments/1ff3ps5/these_5000_... NPM uses at least 5 petabytes per week. 5% of that is 250 terabytes. So $15,000 a week, or $780,000 a year in savings could’ve been gained.
In a great example of the Pareto Principle (80/20), or actually even more extreme, let's only apply this Zopfli optimization if the package download total is equal or more than 1GiB (from the Weekly Traffic in GiB column of the Top 5000 Weekly by Traffic tab of the Google Sheets file from the reddit post). For reference, total bandwidth used by all 5000 packages is 4_752_397 GiB. Packages >= 1GiB bandwidth/week - Tha…
Packages with >= 20GiB bandwidth == 47 packages totaling 2,536,902.81 GiB/week.
Less than 1% of top 5000 packages took 53% of the bandwidth.
5% would be about 127 TiB (rounded up).