Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

61–70 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#61

Post by sltkr »

abound wrote: post #46

A few people have mentioned the environmental angle, but I'd care more about if/how much this slows down decompression on the client. Compressing React 20x slower once is one thing, but 50 million decompressions being even 1% slower is likely net more energy intensive, even accounting for the saved energy transmitting 4-5% fewer bits on the wire.

For formats like deflate, decompression time doesn't generally depend on compressed size. (zstd is similar, though memory use can depend on the compression level used).

This means an optimization like this is virtually guaranteed to be a net positive on the receiving end, since you always save a bit of time/energy when downloading a smaller compressed file.

Re: My failed attempt to shrink all NPM packages by 5%

#63

Post by [deleted] »

coliveira wrote: post #58
liontwist wrote: post #55

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…

The problem is that this guy is treating open source as if it was a company where you need to direct a project to completion. Nobody in open source wants to be told what to do. Just release your work, if it is useful, the community will pick it up and everybody will benefit. You cannot force your improvement into the whole group, even if it is beneficial in your opinion.

[deleted]

Re: My failed attempt to shrink all NPM packages by 5%

#64

Post by Null-Set »

michaelmior wrote: post #59
alt227 wrote: post #10

Earlier quoted context omitted.

I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.

Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.

No, it can't because the checksums won't match.

Re: My failed attempt to shrink all NPM packages by 5%

#65

Post by gregmac »

pif wrote: post #48
jerf wrote: post #29

Earlier quoted context omitted.

"I don't find the cons all that compelling to be honest" This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm. You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to…

> This is a solid example of how things change at scale. 5% is 5% at any scale.

5% off your next lunch and 5% off your next car are very much not the same thing.

Re: My failed attempt to shrink all NPM packages by 5%

#66

Post by hiatus »

michaelmior wrote: post #59
alt227 wrote: post #10

Earlier quoted context omitted.

I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.

Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.

Wouldn't that result in a different checksum for package-lock.json?

Re: My failed attempt to shrink all NPM packages by 5%

#67

Post by dvh »

Imagine being in the middle of nowhere, in winter, on Saturday night, on some farm, knee deep in a cow piss, servicing some 3rd party feed dispenser, only to discover that you have possible solution but it's in some obscure format instead of .tar.gz. Nearest internet 60 miles away. This is what I always imagine happening when some new obscure format come into play, imagine the poor fella, alone, cold, screaming. So incredibly close to his goal, but ultimately stopped by some artificial unnecessary made-up bullshit.

Re: My failed attempt to shrink all NPM packages by 5%

#68

Post by PaulHoule »

Years back I came to the conclusion that conda using bzip2 for compression was a big mistake.

Back then if you wanted to use a particular neural network it was meant for a certain version of Tensorflow which expected you to have a certain version of the CUDA libs.

If you had to work with multiple models the "normal" way to do things was use the developer unfriendly [1][2] installers from NVIDIA to install a single version of the libs at a time.

Turned out you could have many versions of CUDA installed as long as you kept them in different directories and set the library path accordingly, it made sense to pack them up for conda and install them together with everything else.

But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time.

If you were building a new system today you'd probably use zstd since it beats gzip on both speed and compression.

[1] click... click... click...

[2] like they're really going to do something useful with my email address

Re: My failed attempt to shrink all NPM packages by 5%

#69

Post by kittikitti »

Thank you so much for posting this. The original logic was clear and it had me excited! I believe this is useful because compression is very common and although it might not fit perfectly in this scenario, it could very well be a breakthrough in another. If I come across a framework that could also benefit from this compression algorithm, I'll be sure to give you credit.

Re: My failed attempt to shrink all NPM packages by 5%

#70

Post by PaulHoule »

michaelmior wrote: post #60
pif wrote: post #48

Earlier quoted context omitted.

> This is a solid example of how things change at scale. 5% is 5% at any scale.

Yes and no. If I'm paying $5 a month for storage, I probably don't care about saving 5% of my storage costs. If I'm paying $50,000/month in storage costs, 5% savings is a lot more worthwhile to pursue

Doesn't npm belong to Microsoft? It must be hosted in Azure which they own so they must be paying a rock bottom rate for storage, bandwidth, everything.
Post reply on HN