Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

121–130 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#121

Post by vlovich123 »

adgjlsfhk1 wrote: post #47

This seems like a place where the more ambitious version that switches to ZSTD might have better tradeoffs. You would get similar or better compression, with faster decompression and recompression than zstd.It would lose backward compatibility though...

Not necessarily - could retain backward compat by publishing both gzip and zstd variants and having downloaders with newer npm’s prefer to download zstd. Over time, you could require packages only upload zstd going forward and either generate zstd versions of the backlog of unmaintained packages or at least those that see some amount of traffic over some time period if you’re willing to drop very old packages. The ability to install arbitrary versions of packages probably means you’re probably better off reprocessing the backlog although that may cost more than doing nothing.

The package lock checksum is probably a more solvable issue with some coordination.

The benefit of doing this though is less immediate - it will take a few years to show payoff and these kinds of payoffs are not typically made by the kind of committee decisions process described (for better or worse).

Re: My failed attempt to shrink all NPM packages by 5%

#122

Post by TheRealPomax »

cedws wrote: post #25

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

That's on the package publishers, not NPM. They give you an `.npmignore` that's trivially filled out to ensure your package isn't full of garbage, so if someone doesn't bother using that: that's on them, not NPM.

(And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've helped folks reduce their packages by 50+MB in some cases, it's worth your own time as much as it is theirs)

Re: My failed attempt to shrink all NPM packages by 5%

#123

Post by vlovich123 »

fwip wrote: post #76
tyre wrote: post #23

Earlier quoted context omitted.

Size savings translates to latency improvements which directly affects conversion rates. Smaller size isn’t about reducing costs but increased revenue. People care.

Note that this proof-of-concept implementation saves latency on first load, but may add latency at surprising points while using the website. Any user invoking a rarely-used function would see a delay before the javascript executes, without the traditional UI affordances (spinners etc) to indicate that the application was waiting on the network. Further, these secretly-slow paths may change from visit to visit. Many…

> without the traditional UI affordances (spinners etc) to indicate that the application was waiting on the network.

This part is obviously trivially solvable. I think the same basic idea is going to at some point make it but it’ll have to be through explicit annotations first and then there will be tooling to automatically do this for your code based upon historical visits where you get to tune the % of visitors that get additional fetches. Also, you could probably fetch the split off script in the background anyway as a prefetch + download everything rather than just 1 function at a time (or even downloading related groups of functions together)

The idea has lots of merit and you just have to execute it right.

Re: My failed attempt to shrink all NPM packages by 5%

#124

Post by mcherm »

philipwhiuk wrote: post #75
sd9 wrote: post #32

Earlier quoted context omitted.

Yes, but there’s a difference between “this warrants further discussion” and “this warrants further discussion and I’m closing the RFC”. The latter all but guarantees that no further discussion will take place.

No it doesn't. It only does that if you think discussion around future improvements belongs in RFCs.

Where DOES it belong, if not there?

Re: My failed attempt to shrink all NPM packages by 5%

#125

Post by gweinberg »

I was under the impression that bzip compresses more than gzip, but gzip is much faster, so gzip is better for things that need to be compressed on the fly, and bzip is better fro things that get archived. Is this not true? Wouldn't it have been better to use bzip all along for this purpose?

Re: My failed attempt to shrink all NPM packages by 5%

#126

Post by Aicy »

pif wrote: post #48
jerf wrote: post #29

Earlier quoted context omitted.

"I don't find the cons all that compelling to be honest" This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm. You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to…

> This is a solid example of how things change at scale. 5% is 5% at any scale.

That's right, and 5% of a very small number is a very small number. 5% of a very big number is a big number.

Re: My failed attempt to shrink all NPM packages by 5%

#127

Post by ndriscoll »

bhouston wrote: post #84

What about a different approach - an optional npm proxy that recompresses popular packages with 7z/etc in the background? Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready. Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packa…

This felt like the obvious way to do things to me: hash a .tar file, not a .tar.gz file. Use Accept-Encoding to negotiate the compression scheme for transfers. CDN can compress on the fly or optionally cache precompressed files. i.e. just use standard off-the-shelf HTTP features. These days I prefer uncompressed .tar files anyway because ZFS has transparent zstd, so decompressed archive files are generally smaller than a .gz.

Re: My failed attempt to shrink all NPM packages by 5%

#128

Post by dgfitz »

gregmac wrote: post #65
pif wrote: post #48

Earlier quoted context omitted.

> This is a solid example of how things change at scale. 5% is 5% at any scale.

5% off your next lunch and 5% off your next car are very much not the same thing.

So what, instead of 50k for a car you spend 47.5k?

If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it.

5% is 5%.

Re: My failed attempt to shrink all NPM packages by 5%

#129

Post by post-it »

dgfitz wrote: post #128
gregmac wrote: post #65

Earlier quoted context omitted.

5% off your next lunch and 5% off your next car are very much not the same thing.

So what, instead of 50k for a car you spend 47.5k? If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it. 5% is 5%.

I wouldn't pick 5¢ up off the ground but I would certainly pick up $2500.

Re: My failed attempt to shrink all NPM packages by 5%

#130

Post by notpushkin »

aseipp wrote: post #103
1337shadow wrote: post #97

Ok but why doesn't npm registry actually recompress the archives? It can even apply that retroactively, wouldn't require zopli in npm CLI

Hashes of the tarballs are recorded in the package-lock.json of downstream dependants, so recompressing the files in place will cause the hashes to change and break everyone. It has to be done at upload time.

But it still can be done on the npm side, right?
Post reply on HN