Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

91–100 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#91

Post by liontwist »

gjsman-1000 wrote: post #83
liontwist wrote: post #82

Earlier quoted context omitted.

Open source needs to operate differently than a company because people don’t have time/money/energy to deal with bullshit. Hell. Even 15 employees larping as a corporation is going to be inefficient. what you and NPM are telling us, is that they are happy to take free labor, but this is not an open source project. > Engineers are obsessed with 2% optimizations here Actually in large products these are incredible find…

Correct; NPM is not an "open source project" in the sense of a volunteer-first development model. Neither is Linux - over 80% of commits are corporate, and have been for a decade. Neither is Blender anymore - the Blender Development Fund raking in $3M a year calls the shots. Every successful "large" open source project has outgrown the volunteer community. > Actually in large products these are incredible finds. In l…

> breaking compatibility with just 0.1%

Yes. But in this story nothing like that happened.

Re: My failed attempt to shrink all NPM packages by 5%

#92

Post by [deleted] »

liontwist wrote: post #74
coliveira wrote: post #58

Earlier quoted context omitted.

The problem is that this guy is treating open source as if it was a company where you need to direct a project to completion. Nobody in open source wants to be told what to do. Just release your work, if it is useful, the community will pick it up and everybody will benefit. You cannot force your improvement into the whole group, even if it is beneficial in your opinion.

> where you need to direct a project to completion Do you want to get a change in, or not? Is this a project working with the community or not? > Just release your work What motivation exists to optimize package formats if nobody uses that package format? There are no benefits unless it’s in mainline. > Nobody in open source wants to be told what to do He’s not telling anybody to do work. He is sharing an optimizatio…

[deleted]

Re: My failed attempt to shrink all NPM packages by 5%

#93

Post by gjsman-1000 »

liontwist wrote: post #91
gjsman-1000 wrote: post #83

Earlier quoted context omitted.

Correct; NPM is not an "open source project" in the sense of a volunteer-first development model. Neither is Linux - over 80% of commits are corporate, and have been for a decade. Neither is Blender anymore - the Blender Development Fund raking in $3M a year calls the shots. Every successful "large" open source project has outgrown the volunteer community. > Actually in large products these are incredible finds. In l…

> breaking compatibility with just 0.1% Yes. But in this story nothing like that happened.

But NPM has no proof their dashboard won't light up full of corporate customers panicking the moment it goes to production; because their hardcoded integration to have AWS download packages and decompress them with a Lambda and send them to an S3 bucket can no longer decompress fast enough while completing other build steps to avoid mandatory timeouts; just as one stupid example of something that could go wrong. IT is also demanding now that NPM fix it rather than modify the build pipeline which would take weeks to validate, so corporate's begging NPM to fix it by Tuesday's marketing blitz.

Just because it's safe in a lab provides no guarantee it's safe in production.

Re: My failed attempt to shrink all NPM packages by 5%

#94

Post by horsawlarway »

pif wrote: post #48
jerf wrote: post #29

Earlier quoted context omitted.

"I don't find the cons all that compelling to be honest" This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm. You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to…

> This is a solid example of how things change at scale. 5% is 5% at any scale.

5% of newly published packages, with a potentially serious degradation to package publish times for those who have to do that step.

Given his numbers, let's say he saves 100Tb of bandwidth over a year. At AWS egress pricing... that's $5,000 total saved.

And arguably - NPM is getting at least some of that savings by adding CPU costs to publishers at package time.

Feels like... not enough to warrant a risky ecosystem change to me.

Re: My failed attempt to shrink all NPM packages by 5%

#95

Post by snizovtsev »

Yes, but it was expected. It's like prioritising code readability over performance everywhere but the hot path.

Earlier in my career, I managed to use Zopfli once to compress gigabytes of PNG assets into a fast in-memory database supporting a 50K+ RPS web page. We wanted to keep it simple and avoid the complexity of horizontal scaling, and it was OK to drop some rarely used images. So the more images we could pack into a single server, the more coverage we had. In that sense Zopfli was beneficial.

Re: My failed attempt to shrink all NPM packages by 5%

#96

Post by [deleted] »

efitz wrote: post #89
liontwist wrote: post #55

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…

I think that the reason NPM responded this way is because it was a premature optimization. If/when NPM has a problem - storage costs are too high, or transfer costs are too high, or user feedback indicates that users are unhappy with transfer sizes - then they will be ready to listen to this kind of proposal. I think their response was completely rational, especially given a potentially huge impact on compute costs a…

[deleted]

Re: My failed attempt to shrink all NPM packages by 5%

#98

Post by maccard »

efitz wrote: post #89
liontwist wrote: post #55

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…

I think that the reason NPM responded this way is because it was a premature optimization. If/when NPM has a problem - storage costs are too high, or transfer costs are too high, or user feedback indicates that users are unhappy with transfer sizes - then they will be ready to listen to this kind of proposal. I think their response was completely rational, especially given a potentially huge impact on compute costs a…

I disagree with it being a premature optimisation. Treating everything that you haven’t already identified personally as a problem as a premature optimisation is cargo cutting in its own way. The attitude of not caring is why npm and various tools are so slow.

That said, I think NPM’s response was totally correct - explain the problem and the tradeoffs. And OP decided the tradeoffs weren’t worth it, which is totally fair.

Re: My failed attempt to shrink all NPM packages by 5%

#99

Post by avodonosov »

KTibow wrote: post #80
avodonosov wrote: post #13

My experiment on how to reduce javascript size of every web app by 30-50% : https://github.com/avodonosov/pocl Working approach, but in the end I abandoned the project - I doubt people care about such js size savings.

I think this is called tree shaking and Vite/Rollup do this by default these days. Of course, it's easy when you explicitly say what you're importing.

That's not tree-shaking.

Re: My failed attempt to shrink all NPM packages by 5%

#100

Post by [deleted] »

liontwist wrote: post #55

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…

[deleted]
Post reply on HN