Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

141–150 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#141

Post by hinkley »

syncsynchalt wrote: post #138
hinkley wrote: post #111

Earlier quoted context omitted.

I got measurable decreases in deployment time by shrinking the node_modules directory in our docker images. I think people forget that, when you’re copying the same images to dozens and dozens of boxes, any improvement starts to add up to real numbers.

I've not done it, but have you considered using `pnpm` and volume-mounting a shared persistent `pnpm-store` into the containers? It seems like you'd get near-instant npm installs that way.

The only time npm install was on the critical path was hotfixes. It’s definitely worth considering. But I was already deep into doing people giant favors that they didn’t even notice, so I was juggling many other goals. I think the only thank you I got was from the UI lead, who had some soda straw internet connection and this and another thing I did saved him a bunch of hard to recover timeouts.

Re: My failed attempt to shrink all NPM packages by 5%

#143

Post by maccard »

gjsman-1000 wrote: post #110
maccard wrote: post #105

Earlier quoted context omitted.

That’s an argument against making any change to the packaging system ever. “It might break something somewhere” isn’t an argument, it’s a paralysis against change. Improving the edge locality of delivery of npm packages could speed up npm installs. But speeding up npm installs might cause the CI system which is reliant on it for concurrency issues to have a race condition. Does that mean that npm can’t ever make it f…

It is an argument. An age old argument: "If it ain't broke, don't fix it."

This attitude is how in an age with gigabit fiber, 4GB/s hard drive write speed, 8x4 GHz cores with simd instructions it takes 30+ seconds to bundle a handful of files of JavaScript.

Re: My failed attempt to shrink all NPM packages by 5%

#144

Post by ForOldHack »

advisedwang wrote: post #119
sd9 wrote: post #3

The final pro/cons list: https://github.com/npm/rfcs/pull/595#issuecomment-1200480148 I don't find the cons all that compelling to be honest, or at least I think they warrant further discussion to see if there are workarounds (e.g. a choice of compression scheme for a library like typescript, if they would prefer faster publishes). It would have been interesting to see what eventually played out if the author hadn't…

The pros aren't all that compelling either. The npm repo is the only group that this would really be remotely significant for, and there seemed to be no interest. So it doesn't take much of a con to nix a solution to a non-problem.

Every single download, until the end of time is affected: It speeds up the servers, speeds up the updates, saves disk space on the update servers, and saves on bandwidth costs and usage.

Everyone benefits, the only cost is a ultra microscopic time on the front end, and a tiny cost on the client end, and for a very significant number of users, time and money saved. The examples of compression here...

Re: My failed attempt to shrink all NPM packages by 5%

#145

Post by aja12 »

michaelmior wrote: post #59
alt227 wrote: post #10

Earlier quoted context omitted.

I feel massively increasing publish time is a valid reason not to push this though considering such small gains and who the gains apply to.

Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.

> Probably not worth the added complexity, but in theory, the package could be published immediately with the existing compression and then in the background, replaced with the Zopfli-compressed version.

Checksum matters aside, wouldn't that turn the 5% bandwidth savings into an almost double bandwidth increase though? IMHO, considering the complexity to even make it a build time option, the author made the right call.

Re: My failed attempt to shrink all NPM packages by 5%

#146

Post by imoverclocked »

PaulHoule wrote: post #108
cwmma wrote: post #88

Earlier quoted context omitted.

It's probably less about MS and more about the people downloading the packages

For them it is 5% of something tiny.

Maybe, maybe not. If you are on a bandwidth limited connection and you have a bunch of NPM packages to install, 5% of an hour is a few minutes saved. It's likely more than that because long-transfers often need to be restarted.

Re: My failed attempt to shrink all NPM packages by 5%

#147

Post by kemitche »

dgfitz wrote: post #128
gregmac wrote: post #65

Earlier quoted context omitted.

5% off your next lunch and 5% off your next car are very much not the same thing.

So what, instead of 50k for a car you spend 47.5k? If that moves the needle on your ability to purchase the car, you probably shouldn't be buying it. 5% is 5%.

If it takes 1 hour of effort to save 5%:

- Doing 1 hour of effort to save 5% on your $20 lunch is foolhardy for most people. $1/hr is well below US minimum wage. - Doing 1 hour of effort to save 5% on your $50k car is wise. $2500/hr is well above what most people are making at work.

It's not about whether the $2500 affects my ability to buy the car. It's about whether the time it takes me to save that 5% ends up being worthwhile to me given the actual amount saved.

The question is really "given the person-hours it takes to apply the savings, and the real value of the savings, is the savings worth the person-hours spent?"

Re: My failed attempt to shrink all NPM packages by 5%

#148

Post by woadwarrior01 »

kmacdough wrote: post #139
woadwarrior01 wrote: post #117

From the RFC on github[1]. > Zopfli is written in C, which presents challenges. Unless it was added to Node core, the CLI would need to (1) rewrite Zopfli in JS, possibly impacting performance (2) rely on a native module, impacting reliability (3) rely on a WebAssembly module. All of these options add complexity. Wow! Who's going to tell them that V8 is written in C++? :) [1]: https://github.com/npm/rfcs/pull/595

It's not about C per-se, as much as each native compiled dependency creates additional maintenance concerns. Changes to hardware/OS can require a recompile or even fixes. NPM build system already requires a JavaScript runtime, so is already handled as part of existing maintenance. The point is that Zopfli either needs to be rewritten for a platform-agnostic abstraction they already support, or else Zopfli will be add…

> It's not about C per-se, as much as each native compiled dependency creates additional maintenance concerns. Changes to hardware/OS can require a recompile or even fixes.

This is a canard. zopfli is written in portable C and is far more portable than the nodejs runtime. On any hardware/OS combo that one can build the nodejs runtime, they certainly can also build and run zopfli.

Re: My failed attempt to shrink all NPM packages by 5%

#149

Post by PaulHoule »

imoverclocked wrote: post #146
PaulHoule wrote: post #108

Earlier quoted context omitted.

For them it is 5% of something tiny.

Maybe, maybe not. If you are on a bandwidth limited connection and you have a bunch of NPM packages to install, 5% of an hour is a few minutes saved. It's likely more than that because long-transfers often need to be restarted.

A properly working cache and download manager that supports resume goes a long way.

I could never get Docker to work on my ADSL when it was 2 Mbps (FTTN got it up to 20) though it was fine in the Montreal office which had gigabit.

Post reply on HN