Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

191–200 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#191
post #80

Earlier quoted context omitted.

I think this is called tree shaking and Vite/Rollup do this by default these days. Of course, it's easy when you explicitly say what you're importing.

That's not tree-shaking.

Oh, I guess you would need something like dynamic imports to not include uncommonly used functionality in the main bundle

Re: My failed attempt to shrink all NPM packages by 5%

#192

Earlier quoted context omitted.

That's not actually so straightforward. You pay the 10-100x slowdown once on the compressing side, to save 4-5% on every download - which for a popular package one would expect downloads to be in the millions.

The downloads are cached. The build happens on every publish for every CI build.

Most packages don‘t publish every CI build. Some packages, such as Typescript, publish a nightly build once a day. Even then a longer compression time dosen‘t seem too bad.

Re: My failed attempt to shrink all NPM packages by 5%

#193
post #133
post #65

Earlier quoted context omitted.

5% off your next lunch and 5% off your next car are very much not the same thing.

Those lunches could add up to something significant over time. If you're paying $10 per lunch for 10 years, that's $36,500 which is pretty comparable to the cost of a car.

Which is, then, supporting the fact that scale matter, isn't it?

Here the scale of time is larger and does make the 5$ significant, while it isn't at the scale of a few days.

Re: My failed attempt to shrink all NPM packages by 5%

#194
A pro could have been an extra narrative about carbon footprint savings.

I'm surprised it hasn't been raised when talking about saving 2Tb/year only for React. It represents costs, which doesn't seem to be an issue, but also computing power and storage. (Event with a higher/longer computing power due to slower compression, it's done once per version, which isn't really comprable to the amount of downloads anyway)

Hard to calculate the exact saving, but it would represent a smaller CO2 footprint..

Re: My failed attempt to shrink all NPM packages by 5%

#195

Years back I came to the conclusion that conda using bzip2 for compression was a big mistake. Back then if you wanted to use a particular neural network it was meant for a certain version of Tensorflow which expected you to have a certain version of the CUDA libs. If you had to work with multiple models the "normal" way to do things was use the developer unfriendly [1][2] installers from NVIDIA to install a single ve…

>But oh boy was it slow to unpack those bzip2 packages! Since conda had good caching, if you build environments often at all you could be paying more in decompress time than you pay in compression time. For Paper, I'm planning to cache both the wheel archives (so that they're available without recompressing on demand) and unpacked versions (installing into new environments will generally use hard links to the unpacke…

I designed a system which was a lot like uv but written in Python and when I looked at the politics I decided not to go forward with it. (My system also had the problem that it had to be isolated from other Pythons so it would not get its environment trashed, with the ability for software developers to trash their environment I wasn't sure it was a problem that could be 100% solved. uv solved it by not being written in Python. Genius!)

Re: My failed attempt to shrink all NPM packages by 5%

#196

Earlier quoted context omitted.

If it takes 1 hour of effort to save 5%: - Doing 1 hour of effort to save 5% on your $20 lunch is foolhardy for most people. $1/hr is well below US minimum wage. - Doing 1 hour of effort to save 5% on your $50k car is wise. $2500/hr is well above what most people are making at work. It's not about whether the $2500 affects my ability to buy the car. It's about whether the time it takes me to save that 5% ends up bein…

This is something we often do in our house. We talk about things in terms of hours worked rather than price. I think more people should do it.

By that logic I waste time reading books instead of paying someone else to read them for me.

Re: My failed attempt to shrink all NPM packages by 5%

#197
post #25

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

That's on the package publishers, not NPM. They give you an `.npmignore` that's trivially filled out to ensure your package isn't full of garbage, so if someone doesn't bother using that: that's on them, not NPM. (And it's also a little on the folks who install dependencies: if the cruft in a specific library bothers you, hit up the repo and file an issue (or even MR/PR) to get that .npmignore file filled out. I've h…

It's much better to allowlist the files meant to be published using `files` in package.json because you never know what garbage the user has in their folder at the time of publish.

On a typical project with a build step, only a `dist` folder would published.

Re: My failed attempt to shrink all NPM packages by 5%

#198
post #14

I wonder if it would make more sense to pursue Brotli at this point, Node has had it built-in since 10.x so it should be pretty ubiquitous by now. It would require an update to NPM itself though.

+1 to brotli. Newly published packages could use brotli by default, so old ones stay compatible.

Re: My failed attempt to shrink all NPM packages by 5%

#199
I used zopflipng in the past to optimize PNG images. It made sense since there was no better alternative to store lossless image data than the PNG format at the given time in the given environment. Zopfli is awesome when you are locked in on deflate compression. I feel like if the npm folks would want to optimize for smaller package size a better strategy would be switching to some more effective text compression (e.g. bzip2, xz). That would result into a larger file size reduction than 5% for a smaller CPU time increase compared to Zopfli. You would need to come up with some migration strategy though as this change isn't per-se backwards compatible, but that seems manageable if you are in control of the tooling and infrastructure.

Re: My failed attempt to shrink all NPM packages by 5%

#200
post #196

Earlier quoted context omitted.

This is something we often do in our house. We talk about things in terms of hours worked rather than price. I think more people should do it.

By that logic I waste time reading books instead of paying someone else to read them for me.

If you can get the exact same result for less cost (time and money), why not? Things like enjoyment don't factor in since they can't be directly converted into money.
Post reply on HN