Docker to rate limit image pulls • Hacker News Board

Re: Docker to rate limit image pulls

#71

Post by withinboredom » Mon, Aug 24, 2020, 7:33 PM UTC

nickjj wrote: post #54

Does anyone have any good ideas on how the Docker Hub could be monetized in a way that's user friendly and makes sense? AWS, GCP, Azure, DigitalOcean and even GitHub / GitLab all have private container registry offerings. If your stack is on X provider, chances are you're going to use their private registry service instead of using the Docker Hub because you've gone all-in with that provider. That means private repos…

- Keep the 100 pulls for Docker anon users, but make it monthly.

- Every user gets 600 pulls per month for free.

- Pre/post pay per pull or buy a rate plan. Something like $100USD === 10,000 pulls on "pay-as-you-go" and prepay could reduce the cost per pull.

Re: Docker to rate limit image pulls

#72

Post by atomi » Mon, Aug 24, 2020, 7:34 PM UTC

mikepurvis wrote: post #22

throwanem wrote: post #12

Earlier quoted context omitted.

Do pulls for a "latest" tag bypass the cache?

No they don't, but if you do something silly like spin up a bare VM build node, and then pull your environment every time, then obviously you won't get any caching.

Using `docker-machine` You can use an environment variable to pull from a local docker mirror from new bare VMs.

`export ENGINE_REGISTRY_MIRROR=https://mirror.mysite.com`

Re: Docker to rate limit image pulls

#73

Post by greatgib » Mon, Aug 24, 2020, 7:43 PM UTC

We have seen that a thousand time with google products and co.

First, you have unlimited, then you have reasonably large limits. The rational is "limits are needed to reduce abuses and should not impact normal users".

Then, it starts to be mandatory to be authenticated. Again, officially for reducing abuses.

Once everyone has an account, and are used to limitations, free limits are reduced again, little by little, and finally to the point where you need to take the "pro" offer to have a normal usage.

Re: Docker to rate limit image pulls

#74

Post by heavenlyblue » Mon, Aug 24, 2020, 7:46 PM UTC

nrmitchi wrote: post #51

bsamuels wrote: post #47

Earlier quoted context omitted.

This is an excellent point. Trying to find out if docker build --pull without an accompanying blob download will trigger the rate limiter. If it does, then this will definitely be a reason to riot. It will effectively mean that anyone who wants to do more than 200 builds every 6 hours using the "right" way will have to get a docker pro subscription.

It sounds like it definitely does trigger the rate limiter. > There is a small tradeoff – if you pull an image you already have, this is still counted even if you don’t download the layers. I expect we're just going to see a lot more recycling of build nodes once it has "used up it's docker credits".

All reasonable orgs should have had their private docker repo a long time ago.

Everybody else is living the pipe dream where they have externalised their risk and probably deserve the Docker treatment.

Re: Docker to rate limit image pulls

#75

Post by michaelmrose » Mon, Aug 24, 2020, 8:05 PM UTC

AaronFriel wrote: post #57

michaelmrose wrote: post #55

Earlier quoted context omitted.

All these things cost money to run why would it be surprising if they cost money to utilize their resources beyond a point?

You can either monetize the thing that brings you business directly, adversarially impacting users. Or you can monetize something else that's correlated to those costs to subsidize the main use case and keep new user acquisition frictionless. Like NPM charging large businesses with special needs with NPM Enterprise, or GitHub with teams and CI/CD features and GitHub Enterprise, and so on. Docker sold off Docker Enter…

Forgive my unfamiliarity. From the article

>We’ve been getting questions from customers and the community regarding container image layers. We are not counting image layers as part of the pull rate limits. Because we are limiting on manifest requests,

> For example, roughly 30% of all downloads on Hub come from only 1% of our anonymous users.

The limits appear to be 100 pulls per 6 hour time frame per ip address for anon users and twice as much for authenticated users. The least favorable reading of this is to assume a rolling period so lets roll with that. Pun intended.

According to docker whom I imagine is in a better position to evaluate the situation this will impact almost no users. Logically properly caching downloads seems like it would improve local performance. Do you really need, given the possibility to cache downloads, to pull a new image every 1.8 minutes in environments where paying $5 a month for individuals or $25 a month for a team would be prohibitive?

I'm going to assume that orgs manage a variety of one off and recurring expenses. I don't see how this is any different.

What I think is the most salient point is that docker is not a new endeavor. They already have many users. Acquiring new people who consume resources and pay nothing isn't a valuable proposition for them. Why would it be? Do you wish you had more roommates living with you eating your food and paying nothing towards the rent?

Re: Docker to rate limit image pulls

#76

Post by stingraycharles » Mon, Aug 24, 2020, 8:06 PM UTC

AaronFriel wrote: post #43

bsamuels wrote: post #14

There is very little reason for a build node to need to pull 200 images in 6 hours, and here is why: When a machine issues a ``docker build`` command, the program reads the relevant dockerfile to check for any base images that need to be pulled (a la "FROM:") These base images are identified based on the image repository, image name, and image tag. The first thing docker does is it checks its local registry and tries…

CI/CD systems on AWS, Azure, GCP and others might be running on Kubernetes containers (using kaniko, podman, etc) or using Docker-in-Docker, and there isn't a widely supported or in-cloud-platform tool for sharing cached layers. And as pointed out below, even if you are intelligently caching layers, manifest requests count as a pull. As far as I know, no caching proxies exist for Docker that support limiting manifest…

Surprised to see Docker-in-Docker mentioned so deeply down here. It’s an extremely valid way of doing things, and non-trivial to implement a caching layer for.

Re: Docker to rate limit image pulls

#77

Post by viraptor » Mon, Aug 24, 2020, 8:07 PM UTC

Keverw wrote: post #5

I kinda wonder if Docker as a company is struggling. Redhat made Podman which is a compatible replacement, Then there's swarm but apparently that's not recommended and actively developed anymore, then as far as I know they sold off their enterprise clustering product. Seems Kubernetes is the popular thing now even if a bit complex to setup. Wonder what the current business model? Pretty neat idea of using containers,…

> Podman which is a compatible replacement,

Kinda... It doesn't support caching layers for example which makes it very different in practice.

Re: Docker to rate limit image pulls

#78

Post by stingraycharles » Mon, Aug 24, 2020, 8:13 PM UTC

bigmanwalter wrote: post #32

saagarjha wrote: post #24

Earlier quoted context omitted.

> Most of us download our OS via torrents only I mean, I only do it to stick it to the people who claim torrents can only be used for piracy; I think most people prefer the simplicity of direct downloads though…

I can usually max out my connection speed with torrents. This is rarely the case with direct downloads.

The start up time of a torrent is typically so long though, by the time it has connected to all peers and is downloading I have already downloaded the iso with a direct download anyway.

Re: Docker to rate limit image pulls

#79

Post by marcosdumay » Mon, Aug 24, 2020, 8:15 PM UTC

Keverw wrote: post #5

I kinda wonder if Docker as a company is struggling. Redhat made Podman which is a compatible replacement, Then there's swarm but apparently that's not recommended and actively developed anymore, then as far as I know they sold off their enterprise clustering product. Seems Kubernetes is the popular thing now even if a bit complex to setup. Wonder what the current business model? Pretty neat idea of using containers,…

Isn't everybody struggling right now?

Except for Zoom, of course.

Re: Docker to rate limit image pulls

#80

Post by rhacker » Mon, Aug 24, 2020, 8:15 PM UTC

Ugh, I don't envy their position. There are many ways to reduce the size of a docker image. I'm guilty too. Probably the best thing to do is leverage multi-stage builds. Those have the largest effect on repo size. (Like a 10x reduction often).

The problem is, docker, the company behind the repo, has no control over what Open Source Joe and Developer Suzy are committing and the other developers pulling down their images. They can send out all these notices and announcements, and I think the typical reach of such things probably gets like what, %0.05 percent of the developers it needs to?

And of those, are any willing to rewrite the image to be smaller?