This is a nice guy
My failed attempt to shrink all NPM packages by 5%
151–160 of 253 posts
Re: My failed attempt to shrink all NPM packages by 5%
#152Earlier quoted context omitted.
I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.
> The proposal wasn't rejected! They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider ? Please go and run more tests." It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares…
This doesn't seem quite correct to me. They weren't asking for "more validation than was reasonable". They were asking for literally any proof that users would benefit from the proposal. That seems like an entirely reasonable thing to ask before changing the way every single NPM package gets published, ever.
I do agree that 10k/week is non-negligible. Perhaps that means the people responsible for the 10k weren't in the room?
Re: My failed attempt to shrink all NPM packages by 5%
#153> Integrating Zopfli into the npm CLI would be difficult.
Is it possible to modify "gzip -9" or zlib to invoke zopfli? This way everyone who wants to compress better will get the extra compression automatically, in addition to npm.
There will be an increase in compression time, but since "gzip -9" is not the default, people preferring compression speed might not be affected.
Re: My failed attempt to shrink all NPM packages by 5%
#1545% improvement is basically the minimum I usually consider worthwhile to pursue, but it's still small. Once you get to 10% or 20%, things become much more attractive. I can see how people can go either way on a 5% increase if there are any negative consequences (such as increased build time).
Re: My failed attempt to shrink all NPM packages by 5%
#155Imagine being in the middle of nowhere, in winter, on Saturday night, on some farm, knee deep in a cow piss, servicing some 3rd party feed dispenser, only to discover that you have possible solution but it's in some obscure format instead of .tar.gz. Nearest internet 60 miles away. This is what I always imagine happening when some new obscure format come into play, imagine the poor fella, alone, cold, screaming. So i…
I believe zopfli compression is backwards compatible with DEFLATE, it just uses more CPU during the compression phase.
Re: My failed attempt to shrink all NPM packages by 5%
#156Earlier quoted context omitted.
I felt the same. The proposal wasn't rejected! Also, performance gains go beyond user stories - e.g. they reduce infra costs and environmental impact - so I think the main concerns of the maintainers could have been addressed.
> The proposal wasn't rejected! They soft-rejected by requiring more validation than was reasonable. I see this all the time. "But did you consider ? Please go and run more tests." It's pretty clear that the people making the decision didn't actually care about the bandwidth savings, otherwise they would have put the work in themselves to do this, e.g. by requiring Zopfli for popular packages. I doubt Microsoft cares…
Or another way to look at it is it's just (at most!) 5% off an already large bill, and it might cost more than that elsewhere.
And I can buy 225 TB of bandwidth for less than $2k, I assume Microsoft can get better than some HN idiot buying Linode.
Re: My failed attempt to shrink all NPM packages by 5%
#157What about a different approach - an optional npm proxy that recompresses popular packages with 7z/etc in the background? Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready. Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packa…
This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller th…
> hash a .tar file, not a .tar.gz file
For security reasons, it's usually better to hash the compressed file, since it reduces the attack surface: the decompressor is not exposed to unverified data. There have already been vulnerabilities in decompressor implementations which can be exploited through malformed compressed data (and this includes IIRC at least one vulnerability in zlib, which is the standard decompressor for .gz).
Re: My failed attempt to shrink all NPM packages by 5%
#158> Integrating Zopfli into the npm CLI would be difficult. Is it possible to modify "gzip -9" or zlib to invoke zopfli? This way everyone who wants to compress better will get the extra compression automatically, in addition to npm. There will be an increase in compression time, but since "gzip -9" is not the default, people preferring compression speed might not be affected.
You'd have more problems here, but you could do it - if you let it take ages and ages to percolate though all environments.
It's been almost 30 years since bzip2 was released and even now not everything can handle tar.bz2
Re: My failed attempt to shrink all NPM packages by 5%
#159I wonder if there is a way to install the npm packages without the crap they come included with (like docs, tests, readme etc).
Re: My failed attempt to shrink all NPM packages by 5%
#160Earlier quoted context omitted.
So what, instead of 50k for a car you spend 47.5k? If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it. 5% is 5%.
If it takes 1 hour of effort to save 5%: - Doing 1 hour of effort to save 5% on your $20 lunch is foolhardy for most people. $1/hr is well below US minimum wage. - Doing 1 hour of effort to save 5% on your $50k car is wise. $2500/hr is well above what most people are making at work. It's not about whether the $2500 affects my ability to buy the car. It's about whether the time it takes me to save that 5% ends up bein…
This is something we often do in our house. We talk about things in terms of hours worked rather than price. I think more people should do it.