Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

201–210 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#201

Post by mootothemax »

sd9 wrote: post #3

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't…

> I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion

It needs a novel JS port of a C compresison library, which will be wired into a heavily-used and public-facing toolchain, and is something that will ruin a significant number of peoples' days if it breaks.

For me, that kind of ask needs a compelling use case from the start.

Re: My failed attempt to shrink all NPM packages by 5%

#202

Post by pclmulqdq »

ForOldHack wrote: post #144
advisedwang wrote: post #119

Earlier quoted context omitted.

The pros aren't all that compelling either. The npm repo is the only group that this would really be remotely significant for, and there seemed to be no interest. So it doesn't take much of a con to nix a solution to a non-problem.

Every single download, until the end of time is affected: It speeds up the servers, speeds up the updates, saves disk space on the update servers, and saves on bandwidth costs and usage. Everyone benefits, the only cost is a ultra microscopic time on the front end, and a tiny cost on the client end, and for a very significant number of users, time and money saved. The examples of compression here...

Plus a few years of a compression expert writing a JS implementation of what was likely some very cursed C. And someone auditing its security. And someone maintaining it.

Re: My failed attempt to shrink all NPM packages by 5%

#203

Post by ggm »

post-it wrote: post #129
dgfitz wrote: post #128

Earlier quoted context omitted.

So what, instead of 50k for a car you spend 47.5k? If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it. 5% is 5%.

I wouldn't pick 5¢ up off the ground but I would certainly pick up $2500.

You'd keep 5c. A significant number of people who find sums up around $2500 give it back unconditionally, with no expectation of reward. Whoever lost $2500 is having a really bad day.

Re: My failed attempt to shrink all NPM packages by 5%

#204

Post by jay_kyburz »

dgfitz wrote: post #196
jay_kyburz wrote: post #160

Earlier quoted context omitted.

This is something we often do in our house. We talk about things in terms of hours worked rather than price. I think more people should do it.

By that logic I waste time reading books instead of paying someone else to read them for me.

Paying somebody else to read the book means you don't get the benefit of the book.

Also, this is exactly what you company is doing, paying you to "read the book" so they don't have to.

Re: My failed attempt to shrink all NPM packages by 5%

#205

Post by jopsen »

nikeee wrote: post #43

I don't see why it wouldn't be possible to hide behind a flag once Node.js supports zopfli natively. In case of CI/CD, it's totally feasible to just add a --strong-compression flag. In that case, the user expects it to take its time. TS releases a non-preview version every few months, so using 2.5 minutes for compression would work.

Think of the complexity involved, think of having to fix bugs because something went wrong.

Such effort would be better spent preparing npm/node for a future where packages with a lower-bound npm version constraint can be compressed with zstd or similar.

Every feature you add to a program is complexity, you need to really decide if you want it.

Re: My failed attempt to shrink all NPM packages by 5%

#206

Post by MBCook »

Alifatisk wrote: post #39
cedws wrote: post #25

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

At least, switch to pnpm minimize the bloat

As someone who mostly works in Java it continues to floor me that this isn’t the default. Why does every project I work on need an identical copy of possibly hundreds of packages if they’re the same version?

I also like Yarn pnp’s model of leaving node_modules as zip files. CPUs are way faster than storage, they can decompress on the fly. Less disk space at rest, less disk slack, less filesystem bookkeeping.

Every single filesystem is way faster at dealing with one file than dozens/hundreds. Now multiply that by the the hundreds if does, it add up.

Re: My failed attempt to shrink all NPM packages by 5%

#207

Post by vivzkestrel »

I ll give you an even better idea but it ll need atleast a 100 volunteers, maybe a 1000. Take each package and rewrite it without external dependencies. That will cut tech debt for that package significantly. Just like how we have a @types/xyz where some dude named DefinitelyTyped is busy making typescript packages for everything, lets make a namespace like @efficient/cors @efficient/jsdom @efficient/jest etc and eliminate all external dependencies completely for every library on npm

Re: My failed attempt to shrink all NPM packages by 5%

#208

Post by MBCook »

adgjlsfhk1 wrote: post #47

This seems like a place where the more ambitious version that switches to ZSTD might have better tradeoffs. You would get similar or better compression, with faster decompression and recompression than zstd.It would lose backward compatibility though...

Thats a much higher hurdle to jump. I don’t blame the author for trying this first.

If accepted, it might have been a good stepping stone too. A chance to get to know everyone and their concerns and how they think.

So if you wanted to see how this works (proposal + in prod) and then come back later proposing something bigger by switching off zip that would make sense to me as a possible follow up.

Re: My failed attempt to shrink all NPM packages by 5%

#209

Post by MBCook »

frabjoused wrote: post #169

This reminds me of a time I lost an argument with John-David Dalton about cleaning up/minifying lodash as an npm dependency, because when including the readme and license for every sub-library, a lodash import came to ~2.5MB at the time. This also took a lot of seeking time for disks because there were so many individual files. The conversation started and ended at the word cache.

> This also took a lot of seeking time for disks because there were so many individual files.

The fact NPM keeps things in node_modules unzipped seems wild to me. Filesystems are not great at hundreds of thousands of little files. Some are bad, others are terrible.

Zip files are easier to store, take up less space, and CPUs are faster than disks so the decompression in memory is probably faster reading the unzipped files.

That was one of my favorite features of Yarn when I tried it - pnp mode. But since it’s not what NPM does it requires a shim that doesn’t work with all ps mage’s. Or at least didn’t a few years ago.

Re: My failed attempt to shrink all NPM packages by 5%

#210

Post by MBCook »

rafaelmn wrote: post #186
zahlman wrote: post #166

Earlier quoted context omitted.

This seems like a non-starter to me - new packages are added to npm all the time, and will alter the word frequency distribution. If you aren't prepared to re-build constantly and accept that the dictionary isn't optimal, then it's hard to imagine it being significantly better than what you build with a more naive approach. Basically - why try to fine-tune to a moving target?

But is it really moving that fast ? I suspect most fundamental terms in programming and the variations do not change often. You will always have keywords, built ins and the most popular concepts from libs/frameworks. So it is basically downloading a few hundred kb dictionary every year ?

You’d have language keywords and such, yeah.

But past that the most common words in a React project are going to be very different from a Vue project right?

Post reply on HN