Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are Package Registries Holding Open-Source Hostage?

about.scarf.sh

Post reply on HN
Print view View original

71–78 of 78 posts

Re: Are Package Registries Holding Open-Source Hostage?

#71

Post by di »

luhn wrote: post #41
takluyver wrote: post #29

Earlier quoted context omitted.

I agree, though it's worth noting that while volunteers can maintain the software and administer the indexes, they also rely on infrastructure provided by big corporations. E.g. the Python Package Index runs on a CDN provided by fastly, which serves hundreds of TB per day. I very much doubt the non-profit Python Software Foundation could afford that bandwidth if it wasn't an in-kind donation.

It's a couple petabytes, Michael. What could it cost, $10? Seriously though, Fastly's donation of their CDN service is generous and eases the burden on the PSF, but if push came to shove they could definitely afford the bandwidth. In 2018 they had a net income of half a million.

Hi, PyPI maintainer and PSF director here.

There's absolutely no way the PSF could afford PyPI's bandwidth out of pocket. Last I checked our "bill" from Fastly would be close to $1.5M/month.

Also given that PyPI is critical infrastructure for millions of people and software projects, anything cheaper would not really cut it.

Re: Are Package Registries Holding Open-Source Hostage?

#72

Post by LockAndLol »

Qwertious wrote: post #52
LockAndLol wrote: post #51

We have IPFS and the code for hosting most registries is open-source. If the opensource community really wanted to / got annoyed enough, it would devise a system that used those components to make a distributed package registry. It's easy to complain, it's more difficult to work on solutions. We should all be doing more of the latter (working on solutions).

Working on solutions does nothing if you're not working on the right solution. There's nothing quite as useful as a really precise complaint.

I disagree. A precise bug report is good, a "precise complaint" is a mere opinion. It feels really nice to write one and people like patting themselves on their back, but opinions are like assholes, everybody has one.

Personally, I'd much rather see a solution to a problem than a complaint. The solution might involve discussions that go back and forth, but if they culminate in a decision on a way forward with a person willing to do the work, they are much more useful than "your code sucks on line".

Complaining isn't contributing.

Edit: also, working on a solution doesn't preclude discussion on the proper solution.

Re: Are Package Registries Holding Open-Source Hostage?

#73

Post by jka »

edoceo wrote: post #63
im3w1l wrote: post #59

Earlier quoted context omitted.

A 13 year old dabbler cannot self host using their home connection. And that is arguably a problem. Hell I don't know how I would self-host, I just (reluctantly) put stuff on paid-for servers. I guess you start by calling your ISP and asking if they can pretty please give you a static IPV4?

Ok, I get it. I host on a cloud VPS which costs $$/mo - I don't consider hosting on home system a blocker. Even still, you don't need a static. Easy to do DNS map and punch a hole in your FW or have some DMZ. There is not a way to solve for zero-cash and zero-work

You got it, yep; I should've been more precise about what I meant by self-hosting.

Smartphones and data plans would provide the bare necessities for self-hosting; in many cases there's abundant computing and bandwidth resource available to them. Many packages/containers are small and downloaded infrequently - any much data capacity, I expect, goes unused.

If and when smartphone-hosted resources become insufficient, a cloud VPS like your approach would be a next logical upgrade. Beyond that, corporate/foundation-based sponsorship and dedicated servers and bandwidth.

The key would be to make it near-seamless to migrate between those different environments. Namespaced source code repositories and packages appear to have worked well for the likes of GitHub, GitLab, NPM and Docker Hub, so perhaps following similar conceptual design ideas would make sense.

A few areas of concern would be:

- How do you keep end-user devices safe if they will be hosting content for a wide audience?

- How do you react to credentials and other protected content being posted if repositories themselves are a distributed network?

- How do you achieve discoverability and search of content in a distributed environment?

... not to mention whether the effort and migration to such a model is worth the benefits.

I tend to think it would be, since it aligns the incentives around spending, increases hardware utilization, and increases resilience by removing single points of failure.

That said, I also imagine there are well-founded and sincere arguments for continued centralized code and container hosting that are valid and worthwhile (not least of which: it's where we are, and it's relatively straightforward to reason about).

Re: Are Package Registries Holding Open-Source Hostage?

#74

Post by jka »

jka wrote: post #73
edoceo wrote: post #63

Earlier quoted context omitted.

Ok, I get it. I host on a cloud VPS which costs $$/mo - I don't consider hosting on home system a blocker. Even still, you don't need a static. Easy to do DNS map and punch a hole in your FW or have some DMZ. There is not a way to solve for zero-cash and zero-work

You got it, yep; I should've been more precise about what I meant by self-hosting. Smartphones and data plans would provide the bare necessities for self-hosting; in many cases there's abundant computing and bandwidth resource available to them. Many packages/containers are small and downloaded infrequently - any much data capacity, I expect, goes unused. If and when smartphone-hosted resources become insufficient, a…

tl;dr - GitTorrent[1][2], and more recently, radicle[3]

[1] - https://blog.printf.net/articles/2015/05/29/announcing-gitto...

[2] - https://hn.algolia.com/?q=GitTorrent

[3] - https://radicle.xyz/

Re: Are Package Registries Holding Open-Source Hostage?

#75

Post by webology »

takluyver wrote: post #70
luhn wrote: post #41

Earlier quoted context omitted.

It's a couple petabytes, Michael. What could it cost, $10? Seriously though, Fastly's donation of their CDN service is generous and eases the burden on the PSF, but if push came to shove they could definitely afford the bandwidth. In 2018 they had a net income of half a million.

The PSF's tax returns are published on python.org, and 'revenue less expenses' for 2018 was just under $280k. IANA accountant, so maybe that's the wrong line to look at.

Hi, I'm a PSF Director and the PSF's Treasure as of this year. For transparency, our tax returns as of 2018 (and soon 2019) are up on https://www.python.org/psf/records/

You are right that this donation does not show up in our tax filings because they provide it to any OSS project.

Re: Are Package Registries Holding Open-Source Hostage?

#76

Post by autarch »

webology wrote: post #75
takluyver wrote: post #70

Earlier quoted context omitted.

The PSF's tax returns are published on python.org, and 'revenue less expenses' for 2018 was just under $280k. IANA accountant, so maybe that's the wrong line to look at.

Hi, I'm a PSF Director and the PSF's Treasure as of this year. For transparency, our tax returns as of 2018 (and soon 2019) are up on https://www.python.org/psf/records/ You are right that this donation does not show up in our tax filings because they provide it to any OSS project.

Also, it's not income that you would put in your 990s. Similarly, the donor cannot deduct the expense of the donation.

Re: Are Package Registries Holding Open-Source Hostage?

#77

Post by houdinifxtd »

It would be wonderful to keep the service free, but it's also a fair point that the data usage must be tremendous.

Perhaps a balance could be achieved by rate limiting by IP to encourage caching by devs. I'm guessing a fair amount of waste occurs from CI testing. Normally once an end user has the image that should be enough.

Re: Are Package Registries Holding Open-Source Hostage?

#78

Post by webology »

autarch wrote: post #76
webology wrote: post #75

Earlier quoted context omitted.

Hi, I'm a PSF Director and the PSF's Treasure as of this year. For transparency, our tax returns as of 2018 (and soon 2019) are up on https://www.python.org/psf/records/ You are right that this donation does not show up in our tax filings because they provide it to any OSS project.

Also, it's not income that you would put in your 990s. Similarly, the donor cannot deduct the expense of the donation.

Correct on both points. (I do consider it a potential liability though.)
Post reply on HN