81–90 of 253 posts
I mean 4-5% the size for 10-100x the time is not worth it.
That's not actually so straightforward. You pay the 10-100x slowdown once on the compressing side, to save 4-5% on every download - which for a popular package one would expect downloads to be in the millions.
The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…
> What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line. Says who? Says an engineer? Says a product person? NPM is a company with 14 employees; with a system integrated into countless extremely niche and…
Hell. Even 15 employees larping as a corporation is going to be inefficient.
what you and NPM are telling us, is that they are happy to take free labor, but this is not an open source project.
> Engineers are obsessed with 2% optimizations here
Actually in large products these are incredible finds. But ok. They should have the leadership to know which bandwidth tradeoffs they are committed to and tell him immediately it’s not what they want, rather than sending him to various gatekeepers.
Earlier quoted context omitted.
> What’s a little sad, is NPM should not be operating like a company with 1000+ employees. The “persuade us users want this” approach is only going to stop volunteers. They should be proactively identifying efforts like this and helping you bring it across the finish line. Says who? Says an engineer? Says a product person? NPM is a company with 14 employees; with a system integrated into countless extremely niche and…
Open source needs to operate differently than a company because people don’t have time/money/energy to deal with bullshit. Hell. Even 15 employees larping as a corporation is going to be inefficient. what you and NPM are telling us, is that they are happy to take free labor, but this is not an open source project. > Engineers are obsessed with 2% optimizations here Actually in large products these are incredible find…
> Actually in large products these are incredible finds.
In large products, incredible finds may be true; but breaking compatibility with just 0.1% of your customers is also an incredible disaster.
Could verify package integrity by hashing contents rather than archives, plus digital signatures for recompressed versions. Only kicks in for frequently downloaded packages once compression is ready.
Benefits: No npm changes needed, opt-in only, potential for big bandwidth savings on popular packages. Main tradeoff is additional verification steps, but they could be optional given a digital signature approach.
Curious if others see major security holes in this approach?
Also for bigger companies this is not really a "big" problem as they usually have in-house proxies (as you cannot rely on a 3rd party repository in CI/CD for multiple reasons (security, audit, speed, etc)).
The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…
Earlier quoted context omitted.
Yes and no. If I'm paying $5 a month for storage, I probably don't care about saving 5% of my storage costs. If I'm paying $50,000/month in storage costs, 5% savings is a lot more worthwhile to pursue
Doesn't npm belong to Microsoft? It must be hosted in Azure which they own so they must be paying a rock bottom rate for storage, bandwidth, everything.
The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…
If/when NPM has a problem - storage costs are too high, or transfer costs are too high, or user feedback indicates that users are unhappy with transfer sizes - then they will be ready to listen to this kind of proposal.
I think their response was completely rational, especially given a potentially huge impact on compute costs and/or publication latency.
The fact that you are pursuing this is admirable. But this whole thing sounds too much like work. Finding opportunities, convincing entrenched stakeholders, accommodating irrelevant feedback, pitching in meetings — this is the kind of thing that top engineers get paid a lot of money to do. For me personally open source is the time to be creative and free. So my tolerance for anything more than review is very low. And…
...which makes it all the weirder that security isn't any better, as in, publishing a package can be done without a review step on the npm side, for example. I find it strange that they haven't doubled down on enterprise offerings, e.g. creating hosted versions (corporate proxies), reviewed / validated / LTS / certified versions of packages, etc.