Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

171–180 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#171

Post by bhouston »

cesarb wrote: post #157
ndriscoll wrote: post #127

Earlier quoted context omitted.

This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller th…

> hash a .tar file, not a .tar.gz file For security reasons, it's usually better to hash the compressed file, since it reduces the attack surface: the decompressor is not exposed to unverified data. There have already been vulnerabilities in decompressor implementations which can be exploited through malformed compressed data (and this includes IIRC at least one vulnerability in zlib, which is the standard decompress…

This suggests one should just upload a tar rather than a compressed file. Makes sense because one can scan the contents for malicious files without risking a decompressor bug.

BTW npm decompressed all packages anyhow because it lets you view the contents these days on its website.

Re: My failed attempt to shrink all NPM packages by 5%

#172

Post by bhouston »

nopurpose wrote: post #113

It reminds me of an effort to improve docker image format and make it move away from being just a tar file. I can't find links anymore, but it was a pretty clever design, which still couldn't beat dumb tar in efficiency.

Transferring around dumb tar is actually smart because the HTTPS connection can negotiate a compressed version of it to transfer - e.g. gzip, brotli, etc. No need to bake in an unchangable compression format into the standard.

Re: My failed attempt to shrink all NPM packages by 5%

#173

Post by bhouston »

aseipp wrote: post #103
1337shadow wrote: post #97

Ok but why doesn't npm registry actually recompress the archives? It can even apply that retroactively, wouldn't require zopli in npm CLI

Hashes of the tarballs are recorded in the package-lock.json of downstream dependants, so recompressing the files in place will cause the hashes to change and break everyone. It has to be done at upload time.

The hashes of the uncompressed tarballs would be great. Then the HTTP connection can negotiate a compression format for transfer (which can change over time at HTTP itself changes) rather than baking it into the NPM package standard (which is incredibly inflexible.)

Re: My failed attempt to shrink all NPM packages by 5%

#174

Post by JoeAltmaier »

These days technology moves so fast it's hard to keep up. The slowest link in the system is the human being.

That's a strong argument that 'if it isn't broke, don't fix it."

LOts of numbers being thrown around, you add up tiny things enough times you can get a big number. But is npm package download the thing that's tanking the internet? No? Then this is a second- or thirt-order optimization.

Re: My failed attempt to shrink all NPM packages by 5%

#176

Post by bhouston »

ndriscoll wrote: post #127
bhouston wrote: post #84

What about a different approach - an optional npm proxy that recompresses popular packages with 7z/etc in the background? Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready. Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packa…

This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller th…

You are correct. They should be uploading and downloading dumb tar files and let the HTTP connection negotiate the compression method. All hashes should be based on the uncompressed raw tar dump. This would be proper separation of concerns.

Re: My failed attempt to shrink all NPM packages by 5%

#177

Post by arccy »

IshKebab wrote: post #45
n4r9 wrote: post #8

Earlier quoted context omitted.

I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.

> The proposal wasn't rejected! They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider ? Please go and run more tests." It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares…

massively increase the open source github actions bill for runners running longer (compute is generally more expensive) to publish for a small decrease in network traffic (bandwidth is cheap at scale)?

Re: My failed attempt to shrink all NPM packages by 5%

#178

Post by arccy »

bombcar wrote: post #158
omoikane wrote: post #153

> Integrating Zopfli into the npm CLI would be difficult. Is it possible to modify "gzip -9" or zlib to invoke zopfli? This way everyone who wants to compress better will get the extra compression automatically, in addition to npm. There will be an increase in compression time, but since "gzip -9" is not the default, people preferring compression speed might not be affected.

You'd have more problems here, but you could do it - if you let it take ages and ages to percolate though all environments. It's been almost 30 years since bzip2 was released and even now not everything can handle tar.bz2

probably because bzip2 isn't a very good format

Re: My failed attempt to shrink all NPM packages by 5%

#179

Post by arccy »

zahlman wrote: post #166
rafaelmn wrote: post #161

I wonder if you could get better results if you built a dictionary over entire npm. I suspect most common words could easily be reduced to 16k word index. Would be much faster, dictionary would probably fit in cache, can even optimize it in memory for cache prefetch.

This seems like a non-starter to me - new packages are added to npm all the time, and will alter the word frequency distribution. If you aren't prepared to re-build constantly and accept that the dictionary isn't optimal, then it's hard to imagine it being significantly better than what you build with a more naive approach. Basically - why try to fine-tune to a moving target?

would it really change that quickly? you might get significant savings from just having keywords, common variable names, standard library functions

Re: My failed attempt to shrink all NPM packages by 5%

#180

Post by canucker2016 »

true_religion wrote: post #120
horsawlarway wrote: post #94

Earlier quoted context omitted.

5% of newly published packages, with a potentially serious degradation to package publish times for those who have to do that step. Given his numbers, let's say he saves 100Tb of bandwidth over a year. At AWS egress pricing... that's $5,000 total saved. And arguably - NPM is getting at least some of that savings by adding CPU costs to publishers at package time. Feels like... not enough to warrant a risky ecosystem c…

https://www.reddit.com/r/webdev/comments/1ff3ps5/these_5000_... NPM uses at least 5 petabytes per week. 5% of that is 250 terabytes. So $15,000 a week, or $780,000 a year in savings could’ve been gained.

In a great example of the Pareto Principle (80/20), or actually even more extreme, let's only apply this Zopfli optimization if the package download total is equal or more than 1GiB (from the Weekly Traffic in GiB column of the Top 5000 Weekly by Traffic tab of the Google Sheets file from the reddit post).

For reference, total bandwidth used by all 5000 packages is 4_752_397 GiB.

Packages >= 1GiB bandwidth/week - That turns out to be 437 packages (there's a header row, so it's rows 2-438) which uses 4_205_510 GiB.

So 88% of the top 5000 bandwidth is consumed by downloading the top 8.7% (437) packages.

5% is about 210 TiB.

Limiting to the top 100 packages by bandwidth results in 3_217_584 GiB, which is 68% of total bandwidth used by 2% of the total packages.

5% is about 161 TiB.

Post reply on HN