Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

51–60 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#51

Post by bufferoverflow »

adgjlsfhk1 wrote: post #47

This seems like a place where the more ambitious version that switches to ZSTD might have better tradeoffs. You would get similar or better compression, with faster decompression and recompression than zstd.It would lose backward compatibility though...

Brotli and lzo1b have good compression ratios and pretty fast decompression speeds. Compression speed should not matter that much, since you only do it once.

https://quixdb.github.io/squash-benchmark/

There even more obscure options:

https://www.mattmahoney.net/dc/text.html

Re: My failed attempt to shrink all NPM packages by 5%

#52

Post by sd9 »

jerf wrote: post #34
sd9 wrote: post #31

Earlier quoted context omitted.

I agree with everything you said, but it doesn’t contradict my point

I'm saying you probably don't find them compelling because from your point of view, the problems don't look important to you. They don't from my point of view either. But my point of view is the wrong point of view. From their point of view this would be plenty to make me think twice and several times over past that from changing something so deeply fundamental to the system for what is a benefit that nobody who is a…

The problems look important but underexplored

Re: My failed attempt to shrink all NPM packages by 5%

#55

Post by liontwist »

The fact that you are pursuing this is admirable.

But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do.

For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And I would have quit at the first roadblock.

What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line.

Re: My failed attempt to shrink all NPM packages by 5%

#56

Post by web007 »

abound wrote: post #46

A few people have mentioned the environmental angle, but I'd care more about if/how much this slows down decompression on the client. Compressing React 20x slower once is one thing, but 50 million decompressions being even 1% slower is likely net more energy intensive, even accounting for the saved energy transmitting 4-5% fewer bits on the wire.

It's very likely zero or positive impact on the decompression side of things.

Starting with smaller data means everything ends up smaller. It's the same decompression algorithm in all cases, so it's not some special / unoptimized branch of code. It's yielding the same data in the end, so writes equal out plus or minus disk queue fullness and power cycles. It's _maybe_ better for RAM and CPU because more data fits in cache, so less memory is used and the compute is idle less often.

It's relatively easy to test decompression efficiency if you think CPU time is a good proxy for energy usage: go find something like React and test the decomp time of gzip -9 vs zopfli. Or even better, find something similar but much bigger so you can see the delta and it's not lost in rounding errors.

Re: My failed attempt to shrink all NPM packages by 5%

#57

Post by dr-detroit »

liontwist wrote: post #55

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…

[dead]

Re: My failed attempt to shrink all NPM packages by 5%

#58

Post by coliveira »

liontwist wrote: post #55

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…

The problem is that this guy is treating open source as if it was a company where you need to direct a project to completion. Nobody in open source wants to be told what to do. Just release your work, if it is useful, the community will pick it up and everybody will benefit. You cannot force your improvement into the whole group, even if it is beneficial in your opinion.

Re: My failed attempt to shrink all NPM packages by 5%

#59

Post by michaelmior »

alt227 wrote: post #10
sd9 wrote: post #3

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't…

I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.

Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.

Re: My failed attempt to shrink all NPM packages by 5%

#60

Post by michaelmior »

pif wrote: post #48
jerf wrote: post #29

Earlier quoted context omitted.

"I don't find the cons all that compelling to be honest" This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm. You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to…

> This is a solid example of how things change at scale. 5% is 5% at any scale.

Yes and no. If I'm paying $5 a month for storage, I probably don't care about saving 5% of my storage costs. If I'm paying $50,000/month in storage costs, 5% savings is a lot more worthwhile to pursue
Post reply on HN