Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker to rate limit image pulls

docker.com

Post reply on HN
Print view View original

81–90 of 274 posts

Re: Docker to rate limit image pulls

#81

Post by spockz »

rhacker wrote: post #80

Ugh, I don't envy their position. There are many ways to reduce the size of a docker image. I'm guilty too. Probably the best thing to do is leverage multi-stage builds. Those have the largest effect on repo size. (Like a 10x reduction often). The problem is, docker, the company behind the repo, has no control over what Open Source Joe and Developer Suzy are committing and the other developers pulling down their imag…

Well, they could use different rate limits depending on the size of the image. Say if the image size (or the size of the added layers) is in line with what we want for dockers you could offer different rates. Unlimited for images Or people will just add a proxy/imagestream in between instead of directly pulling from docker hub.

Re: Docker to rate limit image pulls

#82

Post by grumple »

solatic wrote: post #58
bsamuels wrote: post #14

There is very little reason for a build node to need to pull 200 images in 6 hours, and here is why: When a machine issues a ``docker build`` command, the program reads the relevant dockerfile to check for any base images that need to be pulled (a la "FROM:") These base images are identified based on the image repository, image name, and image tag. The first thing docker does is it checks its local registry and tries…

> This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again. You're assuming that the set of build nodes is relatively static. Plenty of architectures set up autoscaling for the underlying nodes, that terminate servers that aren't being used and relatively soon enough (tens of minutes, hours) spin up ne…

If you’re using docker for production distribution of images, you should be paying for it. That’s exactly the behavior that creates the need for a limit.

Re: Docker to rate limit image pulls

#83

Post by IceWreck »

young_unixer wrote: post #60
Keverw wrote: post #5

I kinda wonder if Docker as a company is struggling. Redhat made Podman which is a compatible replacement, Then there's swarm but apparently that's not recommended and actively developed anymore, then as far as I know they sold off their enterprise clustering product. Seems Kubernetes is the popular thing now even if a bit complex to setup. Wonder what the current business model? Pretty neat idea of using containers,…

> Redhat made Podman which is a compatible replacement I would actually prefer if they made an incompatible replacement. Docker's CLI is pretty bad in my opinion. I want to use Docker the same way I use a headless virtual machine running an SSH server. I want starting/exiting containers to be independent from their 'main process'. I want to attach/detach whenever I need to and execute arbitrary processes. -- Just use…

have you looked at toolbox? Its by Red Hat and works with podman under the hood

Re: Docker to rate limit image pulls

#84

Post by merb »

solatic wrote: post #63

Want to see if you're affected? kubectl get pods --all-namespaces -ojson | jq -r '.items[].spec | .containers[] // [] += .initContainers[] // [] | .image' | sort | uniq | less ... and look for anything that's not from a private registry that you control. Give people two months to migrate? What a nightmare.

well a shitload of stuff from k8s mostly lies in quay.io or k8s.gcr.io

Re: Docker to rate limit image pulls

#85

Post by chrissoundz »

nickjj wrote: post #54

Does anyone have any good ideas on how the Docker Hub could be monetized in a way that's user friendly and makes sense? AWS, GCP, Azure, DigitalOcean and even GitHub / GitLab all have private container registry offerings. If your stack is on X provider, chances are you're going to use their private registry service instead of using the Docker Hub because you've gone all-in with that provider. That means private repos…

I think there was a lot of potential to be made in the service/support aspect of containerization of applications - configuring appropriate dependencies etc. Now probably not so much - most people have probably painstakingly learnt the knowledge by now. Maybe even some kind of container marketplace.

Otherwise, I don't think it's practical to have docker itself as a commercial product.

Re: Docker to rate limit image pulls

#86

Post by Nullabillity »

saagarjha wrote: post #24
Crazyontap wrote: post #19

Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well download the images too if there was support for it.

> Most of us download our OS via torrents only I mean, I only do it to stick it to the people who claim torrents can only be used for piracy; I think most people prefer the simplicity of direct downloads though…

Docker images already need special handling since you download the layers separately and reassemble them. Going from that to full BitTorrent should be transparent to the users.

In fact, there already exist several implementations of it for Docker![0, 1, 2]

[0]: https://coreos.com/blog/torrent-pulls

[1]: https://d7y.io/en-us/

[2]: https://github.com/uber/kraken

Re: Docker to rate limit image pulls

#87

Post by bpfrh »

nickjj wrote: post #54

Does anyone have any good ideas on how the Docker Hub could be monetized in a way that's user friendly and makes sense? AWS, GCP, Azure, DigitalOcean and even GitHub / GitLab all have private container registry offerings. If your stack is on X provider, chances are you're going to use their private registry service instead of using the Docker Hub because you've gone all-in with that provider. That means private repos…

take over supply checks?

E.g. make base images for programming languages, systems, etc and guarantee security.

I think many small companies would like it a lot better if they got could externalize the cost of running docker images with all their dependencies.

Re: Docker to rate limit image pulls

#88

Post by jhenkens »

heavenlyblue wrote: post #74
nrmitchi wrote: post #51

Earlier quoted context omitted.

It sounds like it definitely does trigger the rate limiter. > There is a small tradeoff – if you pull an image you already have, this is still counted even if you don’t download the layers. I expect we're just going to see a lot more recycling of build nodes once it has "used up it's docker credits".

All reasonable orgs should have had their private docker repo a long time ago. Everybody else is living the pipe dream where they have externalised their risk and probably deserve the Docker treatment.

Agreed. Third party package repositories has been a weak point in our CI, and we put all of them behind a self-hosted proxy that we can manage in our own HA fasion. Turns out we get faster pulls from it, as well as being a good internet-citizen.

Re: Docker to rate limit image pulls

#89

Post by cogman10 »

user5994461 wrote: post #41
Crazyontap wrote: post #19

Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well download the images too if there was support for it.

How would bittorrent work in companies? Only HTTP traffic is allowed and often only when going through the company proxy.

You could have a mixed mode pretty easily. Heck, it would even be pretty efficient.

For your own company, you'd host a swarm that is firewalled in. Then when someone says "I want image xyz" the first thing you do is look for seeders in the swarm for that file. If non exists, then you initiate an http download from docker to get the image.

Now you've got fast distribution with low external network traffic.

Not sure how this would play with Cloud provider pricing, though. I don't believe AWS would be too happy seeing their services turned into BT swarms :)

Re: Docker to rate limit image pulls

#90

Post by AaronFriel »

zxcvbn4038 wrote: post #70
scarface74 wrote: post #50

Earlier quoted context omitted.

Pay $5.00 a month. Docker is a business that deserves to get paid if they offer something valuable.

Well that takes all the fun out of it. Looks like Docker itself has a Dockerhub proxy - https://docs.docker.com/registry/recipes/mirror/ https://hackernoon.com/mirror-cache-dockerhub-locally-for-sp... https://stackoverflow.com/questions/32531048/docker-pull-thr... https://docs.docker.com/registry/configuration/ https://www.google.com/amp/s/ops.tips/amp/gists/aws-s3-priva... If using Alpine, looks like docker-registry…

Per your first link:

> What if the content changes on the Hub?

> When a pull is attempted with a tag, the Registry checks the remote to ensure if it has the latest version of the requested content. Otherwise, it fetches and caches the latest content.

If that causes a manifest pull, it counts as a pull and will be rate limited. Yikes! This could lead to wildly nondeterministic behavior.

Post reply on HN