Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are Package Registries Holding Open-Source Hostage?

about.scarf.sh

Post reply on HN
Print view View original

41–50 of 78 posts

Re: Are Package Registries Holding Open-Source Hostage?

#41

Post by luhn »

takluyver wrote: post #29
autarch wrote: post #27

This article ignores the fact that for many languages, the package repository is maintained as free software by volunteers (sometimes with funding from a foundation). This includes Perl, Python, Ruby, Rust, and many others. NPM is the odd one out, really. I don't think letting one company control a language ecosystem's single package registry is a great idea, for all the reasons that the author notes!

I agree, though it's worth noting that while volunteers can maintain the software and administer the indexes, they also rely on infrastructure provided by big corporations. E.g. the Python Package Index runs on a CDN provided by fastly, which serves hundreds of TB per day. I very much doubt the non-profit Python Software Foundation could afford that bandwidth if it wasn't an in-kind donation.

It's a couple petabytes, Michael. What could it cost, $10?

Seriously though, Fastly's donation of their CDN service is generous and eases the burden on the PSF, but if push came to shove they could definitely afford the bandwidth. In 2018 they had a net income of half a million.

Re: Are Package Registries Holding Open-Source Hostage?

#42

Post by wmf »

twunde wrote: post #38
sp332 wrote: post #35

Earlier quoted context omitted.

I was amazed anyone tried to make a free Docker registry. It's like making a CDN, except instead of individual files, it's for a whole app and all of its dependencies. It's a crazy amount of data for storage and bandwidth.

It's more akin to making and then running a CDN but only charging 10% of customers. I'm sure Docker the company was writing it off as a marketing expense, but if they were running this in one of the public clouds that charge for egress, they were paying out a boatload of money just in egress charges (plus more in storage costs)

making and then running a CDN but only charging 10% of customers

Like CloudFlare? Freemium can be a very successful business model.

Re: Are Package Registries Holding Open-Source Hostage?

#43

Post by ArchOversight »

TuringNYC wrote: post #10
jrockway wrote: post #6

The problem with Docker Hub is that they don't let you choose the pricing model. In the end, bandwidth and CPUs are not free, so someone has to pay. For a while it was VCs hoping for growth, but we all know that giving stuff away for free is not sustainable forever. The problem with Docker Hub's pricing model is there are actually two use cases for Docker Hub. One is where some random entity makes some software -- th…

>> The other model is where some commercial entity wants to distribute software to their users. In that case, they'd be happy to pay for their users to anonymously download it. This has hit us big-time last week -- all our k8s components are hosted on DockerHub and our firm is happy to do publisher-paid pulls, but it seems to not be an option. Instead, every customer is expected to get a paid account (if they pass so…

This comment may provide some hope: https://news.ycombinator.com/item?id=25061202

Re: Are Package Registries Holding Open-Source Hostage?

#44

Post by ArchOversight »

wmf wrote: post #23
mschuster91 wrote: post #20

Earlier quoted context omitted.

As long as you're not doing push stuff: 1) set up a series of N docker registry mirrors in pull-through mode ( https://docs.docker.com/registry/recipes/mirror/ , it's as simple as "docker run --rm --name registry -d -p 5000:5000 -e REGISTRY_STORAGE_DELETE_ENABLED=true -e REGISTRY_PROXY_REMOTEURL= https://registry-1.docker.io -v /mnt/persistentdata/registry:/var/lib/registry registry") 2) expose them on the same domai…

Why is no one talking about this solution?

Cause users still need to update where they are pulling from.

There'd need to be a way within docker to alias to the new URL so that what normally would go to docker hub ends up pulling from the mirror.

Re: Are Package Registries Holding Open-Source Hostage?

#45

Post by diegof79 »

I don’t know the specifics about Docker, but at least for NPM the article got some of the things wrong:

> ..npm, and other comparable registries are incentivized to create lock-in

The author’s conclusion doesn’t match the actions of NPM. There are open source implementations of both client and server, you can run your own NPM registry, and you can change the default registry easily.

> What makes npm’s particular scenario even worse is that they've made it so difficult to use a registry that is not npm

Uh?! I’m using Verdaccio (an open source Npm registry) everyday, the setup is extremely easy. Also yarn uses their own registry by default.

I don’t see where is the “lock in” in NPM.

Edit: typo

Re: Are Package Registries Holding Open-Source Hostage?

#47

Post by jka »

This is an important conversation and I can't help but think that it is repeatedly drawn in predictably valley-minded directions.

As a responsible developer your use case is likely: "I want to install package X, and know that my customers are receiving and installing that package when they perform a build"

The fact that individuals cannot self-host content has been held back by the limitations of DNS (content addressing) and IPv4 (routability) for a long time now.

What I ask is this: if you as a developer were able to self-host the libraries and applications you offer to others -- regardless of whether they're open source or proprietary -- would that not solve most of these perverse incentive situations?

- The bandwidth costs would be yours, but that would allow you to find a charging model that works

- If your software became extremely successful - beyond your own ability to pay for the bandwidth - then the companies and individuals who rely upon your software would be incentivized to step in to foot the bill

- Data about software adoption and usage would be de-rigueur shared with the providers who offer the bandwidth for it

- We would not be reliant (in a community sense, but also in a day-to-day operations sense) on the benevolent albeit often-loss-leading hosting of centralized repositories

Re: Are Package Registries Holding Open-Source Hostage?

#48

Post by mschuster91 »

ArchOversight wrote: post #44
wmf wrote: post #23

Earlier quoted context omitted.

Why is no one talking about this solution?

Cause users still need to update where they are pulling from. There'd need to be a way within docker to alias to the new URL so that what normally would go to docker hub ends up pulling from the mirror.

No they do not, that is the entire beauty of the pull-through mirror. For user code, as long as they keep referencing only to Dockerhub images, nothing needs to be changed (edit: except Gitlab CI configurations using docker:dind, which needs to be informed about the mirror).

The only downside is, as said, that it can't cache third party repos (quay.io comes to mind for people involved in k8s). For these, one has to mess with the resolv.conf and self-signed HTTPS certs for the Docker registry mirror.

Re: Are Package Registries Holding Open-Source Hostage?

#50

Post by hinkley »

908B64B197 wrote: post #13
hinkley wrote: post #8

Earlier quoted context omitted.

There are tools for doing this, but it's a matter of cost and complexity to deal with them. Artifactory seems to have a pretty big chunk of this vertical. It supports a few different repository protocols, so it serves as a bit of a one-stop shop that survives technology changes.

If you are fetching multiple GB of images over the network it kinda make sense.

I think the first wave of artifactory customers was also populated by companies with limited network connectivity. It’s nuts to run a Rails or J2EE project if your company is using a pair of 1MB modems for all traffic, even if the dependencies are relatively small. Branch offices are similarly hamstrung. That was part of Perforce’s customer base as well, since they could run a local proxy for source code.

As you get into CI/CD you start to notice that your upstream repo is occasionally down, because it’s getting in the way of some deadline.

Post reply on HN