Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

21–30 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#21

Post by scott_w »

alt227 wrote: post #10
sd9 wrote: post #3

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't…

I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.

I agree, going from 1 second to 2.5 minutes is a huge negative change, in my opinion. I know publishing a package isn't something you do 10x a day but it's probably a big enough change that, were I doing it, I'd think the publish process is hanging and keep retrying it.

Re: My failed attempt to shrink all NPM packages by 5%

#23

Post by tyre »

bluGill wrote: post #18
dagelf wrote: post #16

Earlier quoted context omitted.

Wdym?? 50% is a big deal

50% size savings isn't important to the people who pay for it. They pay at most pennies for 100% savings (that is somehow all the functionality in zero bytes - not worth anything to those paying the bills)

Size savings translates to latency improvements which directly affects conversion rates. Smaller size isn’t about reducing costs but increased revenue. People care.

Re: My failed attempt to shrink all NPM packages by 5%

#24

Post by macspoofing »

sd9 wrote: post #3

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't…

> I don't find the cons all that compelling to be honest

I found it reasonable.

The 5% improvement was balanced against the cons of increased cli complexity, lack of native JS zopfli implementation, and slower compression .. and 5% just wasn't worth it at the moment - and I agree.

>or at least I think they warrant further discussion

I think that was the final statement.

Re: My failed attempt to shrink all NPM packages by 5%

#26

Post by MortyWaves »

cedws wrote: post #25

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

Totally agree with you. I wish npm did a better job of filtering the crap files out of packages.

Re: My failed attempt to shrink all NPM packages by 5%

#27

Post by swiftcoder »

hartator wrote: post #22

I mean 4-5% the size for 10-100x the time is not worth it.

That's not actually so straightforward. You pay the 10-100x slowdown once on the compressing side, to save 4-5% on every download - which for a popular package one would expect downloads to be in the millions.

Re: My failed attempt to shrink all NPM packages by 5%

#28

Post by rererereferred »

alt227 wrote: post #10
sd9 wrote: post #3

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't…

I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.

Since it's backwards compatible, individual maintainers could enable it in their own pipeline if they don't have issues with the slowdown. It sounds like it could be a single flag in the publish command.

Re: My failed attempt to shrink all NPM packages by 5%

#29

Post by jerf »

sd9 wrote: post #3

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't…

"I don't find the cons all that compelling to be honest"

This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm.

You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to make any changes, until you experience your very own "oh man, I really thought this change was perfectly safe and now my entire customer base is trashed" moment, and then suddenly things like "hey, we need to consider how this affects old signatures and the speed of decompression and just generally whether this is worth the non-zero risks for what are in the end not really that substantial benefits".

I do not say this as the wise Zen guru sitting cross-legged and meditating from a position of being above it all; I say it looking at my own battle scars from the Perfectly Safe things I've pushed out to my customer base, only to discover some tiny little nit caused me trouble. Fortunately I haven't caused any true catastrophes, but that's as much luck as skill.

Attaining the proper balance between moving forward even though it incurs risk and just not changing things that are working is the hardest part of being a software maintainer, because both extremes are definitely bad. Everyone tends to start out in the former situation, but then when they are inevitably bitten it is important not to overcorrect into terrified fear of ever changing anything.

Re: My failed attempt to shrink all NPM packages by 5%

#30

Post by Levitating »

stabbles wrote: post #17
stabbles wrote: post #5

One thing that's excellent about zopfli (apart from being gzip compatible) is how easy it is to bootstrap: git clone https://github.com/google/zopfli.git cc -O2 zopfli/src/zopfli/*.c -lm It just requires a C compiler and linker.

The main downside though, it's impressively slow. Comparing to gzip isn't really worth it. Combine pigz (threaded) with zlib-ng (simd) and you get decent performance. pigz is used in `docker push`. For example, gzipping llvm.tar (624MB) takes less than a second for me: $ time /home/harmen/spack/opt/spack/linux-ubuntu24.04-zen2/gcc-13.2.0/pigz-2.8-5ptdjrmudifhjvhb757ym2bzvgtcsoqc/bin/pigz -k hello.tar real 0m0.779s us…

> 2700x slower

That is impressively slow.

In my opinion even the 28x decrease in performance mentioned would be a no-go. Sure the package saves a few bytes but I don't need my entire pc to grind to a halt every time I publish a package.

Besides, storage is cheap but CPU power draw is not. Imagine the additional CO2 that would have to be produced if this RFC was merged.

> 2 gigabytes of bandwidth per year across all installations

This must be a really rough estimate and I am curious how it was calculated. In any case 2 gigabytes over a year is absolutely nothing. Just my home network can produce a terabyte a day.

Post reply on HN