Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My failed attempt to shrink all NPM packages by 5%

evanhahn.com

Post reply on HN
Print view View original

101–110 of 253 posts

Re: My failed attempt to shrink all NPM packages by 5%

#101

Post by stonemetal12 »

gjsman-1000 wrote: post #78
liontwist wrote: post #55

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…

> What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line. Says who? Says an engineer? Says a product person? NPM is a company with 14 employees; with a system integrated into countless extremely niche and…

NPM is a webservice. They could package the top 10-15 enhancements call it V2. When 98% of traffic is V2 turn V1 off. Repeat every 10 years or so until they work their way into having a good protocol.

Re: My failed attempt to shrink all NPM packages by 5%

#102

Post by maccard »

gjsman-1000 wrote: post #78
liontwist wrote: post #55

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…

> What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line. Says who? Says an engineer? Says a product person? NPM is a company with 14 employees; with a system integrated into countless extremely niche and…

> Engineers are obsessed with 2% optimizations here, 5% optimizations there; unchecked, it will literally turn into an OCD outlet, all for things nobody in the non-tech world even notices, let alone asks about. Just let it go.

I absolutely disagree with you. If the world took more of those 5% optimisations here and there everything would be faster. I think more people should look at those 5% optimisations. In many cases they unlock knowledge that results in a 20% speed up later down the line. An example from my past - I was tasked with reducing the running speed of a one shot tool we were using at $JOB. It was taking about 15 minutes to run. I shaved off seconds here and there with some fine grained optimisations, and tens of seconds with some modernisation of some core libraries. Nothing earth shattering but improvements none the less. One day, I noticed a pattern was repeating and I was fixing an issue for the third time in a different place (searching a gigantic array of stuff for a specific entry). I took a step back and realised that if I replaced the mega list with a hash table it might fix every instance of this issue in our app. It was a massive change, touching pretty much every file. And all of a sudden our 15 minute runtime was under 30 seconds.

People used this tool every day, it was developed by a team of engineers wildly smarter than me. But it had grown and nobody really understood the impact of the growth. When it started that array was 30, 50 entries. On our project it was 300,000 and growing every day.

Not paying attention to these things causes decay and rot. Not every change should be taken, but more people should care.

Re: My failed attempt to shrink all NPM packages by 5%

#103

Post by aseipp »

1337shadow wrote: post #97

Ok but why doesn't npm registry actually recompress the archives? It can even apply that retroactively, wouldn't require zopli in npm CLI

Hashes of the tarballs are recorded in the package-lock.json of downstream dependants, so recompressing the files in place will cause the hashes to change and break everyone. It has to be done at upload time.

Re: My failed attempt to shrink all NPM packages by 5%

#104

Post by liontwist »

gjsman-1000 wrote: post #93
liontwist wrote: post #91

Earlier quoted context omitted.

> breaking compatibility with just 0.1% Yes. But in this story nothing like that happened.

But NPM has no proof their dashboard won't light up full of corporate customers panicking the moment it goes to production; because their hardcoded integration to have AWS download packages and decompress them with a Lambda and send them to an S3 bucket can no longer decompress fast enough while completing other build steps to avoid mandatory timeouts; just as one stupid example of something that could go wrong. IT i…

Ok, but why is the burden on him to show that? Are they not interested in improving bandwidth and speed for their users?

The conclusion of this line of reasoning is to never make any change.

If contributions are not welcome, don’t pretend they are and waste my time.

> can no longer decompress fast enough

Already discussed this in another thread. It’s not an issue.

Re: My failed attempt to shrink all NPM packages by 5%

#105

Post by maccard »

gjsman-1000 wrote: post #93
liontwist wrote: post #91

Earlier quoted context omitted.

> breaking compatibility with just 0.1% Yes. But in this story nothing like that happened.

But NPM has no proof their dashboard won't light up full of corporate customers panicking the moment it goes to production; because their hardcoded integration to have AWS download packages and decompress them with a Lambda and send them to an S3 bucket can no longer decompress fast enough while completing other build steps to avoid mandatory timeouts; just as one stupid example of something that could go wrong. IT i…

That’s an argument against making any change to the packaging system ever. “It might break something somewhere” isn’t an argument, it’s a paralysis against change. Improving the edge locality of delivery of npm packages could speed up npm installs. But speeding up npm installs might cause the CI system which is reliant on it for concurrency issues to have a race condition. Does that mean that npm can’t ever make it faster either?

Re: My failed attempt to shrink all NPM packages by 5%

#106

Post by hinkley »

cedws wrote: post #25

Last I checked npm packages were full of garbage including non-source code. There's no reason for node_modules to be as big as it usually is, text compresses extremely well. It's just general sloppiness endemic to the JavaScript ecosystem.

I believe I knocked 10% off of our node_modules directory by filing .npmignore PRs or bug reports to tools we used.

Now if rxjs weren’t a dumpster fire…

Re: My failed attempt to shrink all NPM packages by 5%

#107

Post by knighthack »

pif wrote: post #48
jerf wrote: post #29

Earlier quoted context omitted.

"I don't find the cons all that compelling to be honest" This is a solid example of how things change at scale. Concerns I wouldn't even think about for my personal website become things I need to think about for the download site being hit by 50,000 of my customers become big deals when operating at the scale of npm. You'll find those arguments the pointless nitpicking of entrenched interests who just don't want to…

> This is a solid example of how things change at scale. 5% is 5% at any scale.

Do you even know how absolute numbers work vis-à-vis percentages?

Re: My failed attempt to shrink all NPM packages by 5%

#108

Post by PaulHoule »

cwmma wrote: post #88
PaulHoule wrote: post #70

Earlier quoted context omitted.

Doesn't npm belong to Microsoft? It must be hosted in Azure which they own so they must be paying a rock bottom rate for storage, bandwidth, everything.

It's probably less about MS and more about the people downloading the packages

For them it is 5% of something tiny.

Re: My failed attempt to shrink all NPM packages by 5%

#109

Post by lyu07282 »

gjsman-1000 wrote: post #78
liontwist wrote: post #55

The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…

> What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line. Says who? Says an engineer? Says a product person? NPM is a company with 14 employees; with a system integrated into countless extremely niche and…

> Says who?

> Says an engineer?

I prevent cross-site scripting, I monitor for DDoS attacks, emergency database rollbacks, and faulty transaction handlings. The Internet heard of it? Transfers half a petabyte of data every minute. Do you have any idea how that happens? All those YouPorn ones and zeroes streaming directly to your shitty, little smart phone day after day? Every dipshit who shits his pants if he can't get the new dubstep Skrillex remix in under 12 seconds? It's not magic, it's talent and sweat. People like me, ensuring your packets get delivered, un-sniffed. So what do I do? I make sure that one bad config on one key component doesn't bankrupt the entire fucking company. That's what the fuck I do.

Re: My failed attempt to shrink all NPM packages by 5%

#110

Post by gjsman-1000 »

maccard wrote: post #105
gjsman-1000 wrote: post #93

Earlier quoted context omitted.

But NPM has no proof their dashboard won't light up full of corporate customers panicking the moment it goes to production; because their hardcoded integration to have AWS download packages and decompress them with a Lambda and send them to an S3 bucket can no longer decompress fast enough while completing other build steps to avoid mandatory timeouts; just as one stupid example of something that could go wrong. IT i…

That’s an argument against making any change to the packaging system ever. “It might break something somewhere” isn’t an argument, it’s a paralysis against change. Improving the edge locality of delivery of npm packages could speed up npm installs. But speeding up npm installs might cause the CI system which is reliant on it for concurrency issues to have a race condition. Does that mean that npm can’t ever make it f…

It is an argument. An age old argument:

"If it ain't broke, don't fix it."

Post reply on HN