Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are Package Registries Holding Open-Source Hostage?

about.scarf.sh

Post reply on HN
Print view View original

61–70 of 78 posts

Re: Are Package Registries Holding Open-Source Hostage?

#61

Post by tedivm »

luhn wrote: post #41
takluyver wrote: post #29

Earlier quoted context omitted.

I agree, though it's worth noting that while volunteers can maintain the software and administer the indexes, they also rely on infrastructure provided by big corporations. E.g. the Python Package Index runs on a CDN provided by fastly, which serves hundreds of TB per day. I very much doubt the non-profit Python Software Foundation could afford that bandwidth if it wasn't an in-kind donation.

It's a couple petabytes, Michael. What could it cost, $10? Seriously though, Fastly's donation of their CDN service is generous and eases the burden on the PSF, but if push came to shove they could definitely afford the bandwidth. In 2018 they had a net income of half a million.

As of March PyPi was pushing out 300TB a day through its CDN. Ignoring the "off the shelf price" of $0.12/gb and assuming they negotiate a bulk discount driving them down to $0.05/gb that's still $15,000 a day (or just shy of $5.5 million a year). Their net income in 2018 would cover less than 10% of that bill.

Re: Are Package Registries Holding Open-Source Hostage?

#62

Post by true_religion »

tedivm wrote: post #61
luhn wrote: post #41

Earlier quoted context omitted.

It's a couple petabytes, Michael. What could it cost, $10? Seriously though, Fastly's donation of their CDN service is generous and eases the burden on the PSF, but if push came to shove they could definitely afford the bandwidth. In 2018 they had a net income of half a million.

As of March PyPi was pushing out 300TB a day through its CDN. Ignoring the "off the shelf price" of $0.12/gb and assuming they negotiate a bulk discount driving them down to $0.05/gb that's still $15,000 a day (or just shy of $5.5 million a year). Their net income in 2018 would cover less than 10% of that bill.

Well.... they couldn't afford Fast.ly, or Akamai, or Brightcove, but they could afford any tier-2 or tier-3 CDN.

For example CDN77, would start you off at 0.016/gb if you have more than 100TB per month, without any negotiation.

If you have 300TB per day, you surely can negotiate sub-cent pricing somewhere.

No, the costs are manageable.

I think the charitable/community projects setup for developers, need to act like non-profits in other sectors and actively seek out donations.

Re: Are Package Registries Holding Open-Source Hostage?

#63

Post by edoceo »

im3w1l wrote: post #59
edoceo wrote: post #57

Earlier quoted context omitted.

How can we not self-host? For lots of these you can if you want. I mean, I host docker images and various packages that folks source from our infrastructure. I don't see DNS or IPv4 as a blocker

A 13 year old dabbler cannot self host using their home connection. And that is arguably a problem. Hell I don't know how I would self-host, I just (reluctantly) put stuff on paid-for servers. I guess you start by calling your ISP and asking if they can pretty please give you a static IPV4?

Ok, I get it. I host on a cloud VPS which costs $$/mo - I don't consider hosting on home system a blocker.

Even still, you don't need a static. Easy to do DNS map and punch a hole in your FW or have some DMZ.

There is not a way to solve for zero-cash and zero-work

Re: Are Package Registries Holding Open-Source Hostage?

#64

Post by a1369209993 »

Animats wrote: post #24

Yes, finding a place where you can cut off someone's air supply is a good business model.

More, to the point, a even better business model is to make such a place, manipulate people into it, then pretend you're just charging fair market price for air.

Re: Are Package Registries Holding Open-Source Hostage?

#65

Post by realquadrant »

emilsedgh wrote: post #32
jzb wrote: post #21

FTA: "Ultimately, package registries need to align their incentives with those of maintainers." Putting it all on the registries to come up with a viable business model and provide this to maintainers without any responsibility[1] on the part of the maintainer seems really one-sided. It costs quite a bit of money to run something like Docker Hub or NPM. If you want something aligned software maintainers first and for…

It costs quite a bit of money to run something like Docker Hub or NPM. If you want something aligned software maintainers first and foremost, you want a non-profit / foundation that's got priorities aligned with the larger community and not a for-profit entity that has to justify keeping the lights on. This is an interesting thought. Linux Foundation has a good corporate backing. FSF traditionally provided the backbo…

What do you see as pros and cons of having the Linux Foundation running it and what are the core features?

Re: Are Package Registries Holding Open-Source Hostage?

#66

Post by im3w1l »

edoceo wrote: post #63
im3w1l wrote: post #59

Earlier quoted context omitted.

A 13 year old dabbler cannot self host using their home connection. And that is arguably a problem. Hell I don't know how I would self-host, I just (reluctantly) put stuff on paid-for servers. I guess you start by calling your ISP and asking if they can pretty please give you a static IPV4?

Ok, I get it. I host on a cloud VPS which costs $$/mo - I don't consider hosting on home system a blocker. Even still, you don't need a static. Easy to do DNS map and punch a hole in your FW or have some DMZ. There is not a way to solve for zero-cash and zero-work

From what I heard, hosting on dynamic IP is unreliable.

Re: Are Package Registries Holding Open-Source Hostage?

#67

Post by stubish »

jzb wrote: post #21

FTA: "Ultimately, package registries need to align their incentives with those of maintainers." Putting it all on the registries to come up with a viable business model and provide this to maintainers without any responsibility[1] on the part of the maintainer seems really one-sided. It costs quite a bit of money to run something like Docker Hub or NPM. If you want something aligned software maintainers first and for…

A properly designed registry doesn't have to cost much to run. The costs are incurred when people use the registry for distribution, or for all the infrastructure needed to monetize the registry and track users. All a registry needs to provide is an index of URLs, signing keys, and some useful metadata to enable discovery. Trivial in its purest form, with more cost if you want advanced features like ratings or curation or your own authentication service.

Re: Are Package Registries Holding Open-Source Hostage?

#69

Post by takluyver »

true_religion wrote: post #62
tedivm wrote: post #61

Earlier quoted context omitted.

As of March PyPi was pushing out 300TB a day through its CDN. Ignoring the "off the shelf price" of $0.12/gb and assuming they negotiate a bulk discount driving them down to $0.05/gb that's still $15,000 a day (or just shy of $5.5 million a year). Their net income in 2018 would cover less than 10% of that bill.

Well.... they couldn't afford Fast.ly, or Akamai, or Brightcove, but they could afford any tier-2 or tier-3 CDN. For example CDN77, would start you off at 0.016/gb if you have more than 100TB per month, without any negotiation. If you have 300TB per day , you surely can negotiate sub-cent pricing somewhere. No, the costs are manageable. I think the charitable/community projects setup for developers, need to act like…

I assume there's some difference in e.g. speed or reliability between Fastly (8 c/GB) and CDN77 (1.6 c/GB)? Even if you can negotiate it to 0.8 c/GB, you're still talking about a huge piece of the PSF's budget. They would have to either cut other expenditure (e.g. making grants) dramatically, or find a lot of new income (continuously, not just a one-off donation drive). And PyPI's bandwidth is growing rapidly [1].

If PyPI didn't have sponsor providing bandwidth, I'd guess it would implement some form of rate limiting and encourage people to mirror/cache packages much more to reduce load. I don't think it would die completely, but it would be less convenient and still cost the PSF a fair bit of money.

[1] https://twitter.com/di_codes/status/1235707819955032069

Re: Are Package Registries Holding Open-Source Hostage?

#70

Post by takluyver »

luhn wrote: post #41
takluyver wrote: post #29

Earlier quoted context omitted.

I agree, though it's worth noting that while volunteers can maintain the software and administer the indexes, they also rely on infrastructure provided by big corporations. E.g. the Python Package Index runs on a CDN provided by fastly, which serves hundreds of TB per day. I very much doubt the non-profit Python Software Foundation could afford that bandwidth if it wasn't an in-kind donation.

It's a couple petabytes, Michael. What could it cost, $10? Seriously though, Fastly's donation of their CDN service is generous and eases the burden on the PSF, but if push came to shove they could definitely afford the bandwidth. In 2018 they had a net income of half a million.

The PSF's tax returns are published on python.org, and 'revenue less expenses' for 2018 was just under $280k. IANA accountant, so maybe that's the wrong line to look at.
Post reply on HN