Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker to rate limit image pulls

docker.com

Post reply on HN
Print view View original

51–60 of 274 posts

Re: Docker to rate limit image pulls

#51

Post by nrmitchi »

bsamuels wrote: post #47
nrmitchi wrote: post #38

Earlier quoted context omitted.

I don't disagree with what you are saying, _but_: > Docker defines pull rate limits as the number of manifest requests to Docker Hub. > For example, if you already have the image, the Docker Engine client will issue a manifest request, realize it has all of the referenced layers based on the returned manifest, and stop. ... ... So an image pull is actually one or two manifest requests, This still implies that even if…

This is an excellent point. Trying to find out if docker build --pull without an accompanying blob download will trigger the rate limiter. If it does, then this will definitely be a reason to riot. It will effectively mean that anyone who wants to do more than 200 builds every 6 hours using the "right" way will have to get a docker pro subscription.

It sounds like it definitely does trigger the rate limiter.

> There is a small tradeoff – if you pull an image you already have, this is still counted even if you don’t download the layers.

I expect we're just going to see a lot more recycling of build nodes once it has "used up it's docker credits".

Re: Docker to rate limit image pulls

#52

Post by draw_down »

theon144 wrote: post #15

Good! I'd even welcome much more agressive limits than what they're proposing; the current culture regarding builds and CI in general is horrifyingly ineffficient, wasteful and in the end just plain slow. I'm looking forward to developers adjusting their workflows (and caches, etc.) to actual, reasonable limits, not just perusing the service as if it were an unlimited cost-free cornucopia of software.

> the current culture regarding builds and CI in general is horrifyingly ineffficient

I used to be in charge of the website for a company you’ve heard of. We once realized some huge proportion of our traffic originated from a hosted CI company requesting the site thousands and thousands of times (guessing one for each build they hosted) every 5 minutes.

I can’t remember what proportion of traffic it was but I’m pretty sure it was a majority, maybe even more than 80%.

Re: Docker to rate limit image pulls

#53

Post by michaelmrose »

AaronFriel wrote: post #2

This seems incredibly likely to break development use cases at both extremes: CI/CD systems and developers just starting out could end up pulling quite a few images per hour. Imagine if NPM, Ruby Gems, and so on rate limited package downloads until you paid! I'm not sure if there's a better way to monetize the Docker Hub, but this seems so hostile to adoption.

Presumably companies that pay 5-6 figure salaries to devs can probably pay those devs to set up caching or pay docker for the privilege of not bothering. The team pro plan with no limits starts at $25 a month for 5 users.

Re: Docker to rate limit image pulls

#54

Post by nickjj »

Does anyone have any good ideas on how the Docker Hub could be monetized in a way that's user friendly and makes sense?

AWS, GCP, Azure, DigitalOcean and even GitHub / GitLab all have private container registry offerings.

If your stack is on X provider, chances are you're going to use their private registry service instead of using the Docker Hub because you've gone all-in with that provider. That means private repos alone isn't enough to get folks to pay for Docker Hub.

Re: Docker to rate limit image pulls

#55

Post by michaelmrose »

AaronFriel wrote: post #44
sitkack wrote: post #7

Earlier quoted context omitted.

> Imagine if NPM, Ruby Gems, and so on rate limited package downloads until you paid! Sounds like an entirely reasonable thing to start imagining. Reliability, safety, determinism and predictability are not thrust upon someone from the commons. I frankly find it someone atrocious and abusive that downstream systems do not adequately cache these assets. The main archive repositories should be the source of truth, but…

Do you think NPM, Ruby Gems, or others would last long if they were so user hostile? Would Node have grown into the genuinely useful development environment it is if free users were limited to 10000 package downloads an hour? It's so unbelievably user hostile that it seems like the result will be people just stop using Docker. The right solution for Docker is probably to spin off or monetize the Hub in a different wa…

All these things cost money to run why would it be surprising if they cost money to utilize their resources beyond a point?

Re: Docker to rate limit image pulls

#56

Post by znpy »

user5994461 wrote: post #41
Crazyontap wrote: post #19

Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well download the images too if there was support for it.

How would bittorrent work in companies? Only HTTP traffic is allowed and often only when going through the company proxy.

The company proxy would have to be modified to allow torrent traffic, I guess?

Re: Docker to rate limit image pulls

#57

Post by AaronFriel »

michaelmrose wrote: post #55
AaronFriel wrote: post #44

Earlier quoted context omitted.

Do you think NPM, Ruby Gems, or others would last long if they were so user hostile? Would Node have grown into the genuinely useful development environment it is if free users were limited to 10000 package downloads an hour? It's so unbelievably user hostile that it seems like the result will be people just stop using Docker. The right solution for Docker is probably to spin off or monetize the Hub in a different wa…

All these things cost money to run why would it be surprising if they cost money to utilize their resources beyond a point?

You can either monetize the thing that brings you business directly, adversarially impacting users.

Or you can monetize something else that's correlated to those costs to subsidize the main use case and keep new user acquisition frictionless. Like NPM charging large businesses with special needs with NPM Enterprise, or GitHub with teams and CI/CD features and GitHub Enterprise, and so on.

Docker sold off Docker Enterprise. Now they're trying to extract rents from people who are just trying to "docker build", a command whose ease of use is what drove people to use Docker in the first place.

Foot, meet gun.

Docker is breaking their main selling point. "docker build" and "docker run" should _just work_. By breaking that expectation in subtle ways they risk alienating users. By advertising that they're willing to break their main use case, they're going to alienate businesses and early adopter developers like myself.

Now I'm looking for alternative registries and making sure that devops code I manage doesn't depend on Docker. That's surely not what they wanted, right?

Re: Docker to rate limit image pulls

#58

Post by solatic »

bsamuels wrote: post #14

There is very little reason for a build node to need to pull 200 images in 6 hours, and here is why: When a machine issues a ``docker build`` command, the program reads the relevant dockerfile to check for any base images that need to be pulled (a la "FROM:") These base images are identified based on the image repository, image name, and image tag. The first thing docker does is it checks its local registry and tries…

> This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again.

You're assuming that the set of build nodes is relatively static.

Plenty of architectures set up autoscaling for the underlying nodes, that terminate servers that aren't being used and relatively soon enough (tens of minutes, hours) spin up new servers to replace them as needed.

Rarely do the machine images used to spin up new servers include the base images of the containers that will be spun up to replace them. Much more often, the base machine image is a base OS image, and container images are downloaded on-the-fly as needed. Essentially, the engineering cost of making image-launching more efficient was externalized onto an external provider willing to pay the price.

Re: Docker to rate limit image pulls

#59

Post by jopsen »

Pet_Ant wrote: post #35
adrianmsmith wrote: post #17

Earlier quoted context omitted.

> The first thing docker does is it checks its local registry and tries to find a match for the base image the docker build is requesting. If a matching image is located in the local registry, it uses that one in lieu of downloading the image. While I agree that this is the way it's supposed to work, I have unfortunately worked at companies with "stateless" build/CI servers that download the Docker image each build.

> While I agree that this is the way it's supposed to work, I have unfortunately worked at companies with "stateless" build/CI servers that download the Docker image each build. Couldn't they remain stateless but be redirected through a caching proxy? Memoization is not contrary to statelessness.

Sure, now they have to build a proxy...

Re: Docker to rate limit image pulls

#60

Post by young_unixer »

Keverw wrote: post #5

I kinda wonder if Docker as a company is struggling. Redhat made Podman which is a compatible replacement, Then there's swarm but apparently that's not recommended and actively developed anymore, then as far as I know they sold off their enterprise clustering product. Seems Kubernetes is the popular thing now even if a bit complex to setup. Wonder what the current business model? Pretty neat idea of using containers,…

> Redhat made Podman which is a compatible replacement

I would actually prefer if they made an incompatible replacement. Docker's CLI is pretty bad in my opinion.

I want to use Docker the same way I use a headless virtual machine running an SSH server. I want starting/exiting containers to be independent from their 'main process'. I want to attach/detach whenever I need to and execute arbitrary processes.

-- Just use /bin/bash as the main process

This seems to be the workaround, but I always have problems with containers exiting when I don't want them to and it's just harder than what it needs to be. I've spent a total of like 6 hours learning Docker and I still don't know exactly how to achieve this simple workflow without my containers quitting on me or attach/detach issues. With VirtualBox I can do this easily. Am I too stupid to use Docker?

-- Then just use VirtualBox

That's what I do, but I would like not to have the overhead of a vm.

Post reply on HN