Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker to rate limit image pulls

docker.com

Post reply on HN
Print view View original

261–270 of 274 posts

Re: Docker to rate limit image pulls

#261

Post by hamiltont »

manquer wrote: post #213
oppositelock wrote: post #125

Earlier quoted context omitted.

I run tens of thousands of docker images in production, or rather, tens of thousands of copies of a few hundred images. If you do something like this, you absolutely MUST have a local registry. Harbor [1], JFrog [2], and Quay [3] would be the first ones that I look at. Harbor is open source, free, and a member of the CNCF. You will need to do a little bit of work to set it up to scale properly. JFrog offers a SaaS re…

Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best to use the corresponding registry. I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat. Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mir…

FYI, Using ECR with Docker Swarm is something we did try. It was hellish. We never nailed down the exact problems, but we spent about a month with 2-3 experienced engineers trying to fix the edge case issues.

The main issue was ECR has a slightly different authentication model than docker swarm. The whole '--with-registry-auth' only partially works when you are using ECR. Unfortunately, it works just enough that you think it's working, until all your tokens time out and a worker can suddenly no longer pull an image.

Our common failure case was an image becoming unhealthy or a node being drained. When that image would try to be restarted on a different worker, if that worker did not have the image it would try to get it from the registry. If the tokens were expired it would fail.

The only "fix" we ever found was to setup a cron job that forcibly deployed a new version of a "replicated globally" image every X minutes (where X was based on ECR token expiration). It kind of worked, but we still had occasional failures we could not identify.

I wish it worked better, because it was nice to use ECR. Frankly token expiration sounds much more secure too, but without direct support for token refresh inside the docker engine it's just hard to get everything to work

Re: Docker to rate limit image pulls

#262

Post by [deleted] »

manquer wrote: post #213
oppositelock wrote: post #125

Earlier quoted context omitted.

I run tens of thousands of docker images in production, or rather, tens of thousands of copies of a few hundred images. If you do something like this, you absolutely MUST have a local registry. Harbor [1], JFrog [2], and Quay [3] would be the first ones that I look at. Harbor is open source, free, and a member of the CNCF. You will need to do a little bit of work to set it up to scale properly. JFrog offers a SaaS re…

Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best to use the corresponding registry. I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat. Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mir…

[deleted]

Re: Docker to rate limit image pulls

#263

Post by hamiltont »

oppositelock wrote: post #125
hamiltont wrote: post #110

Earlier quoted context omitted.

Do you run a local registry? Any high-quality articles/youtube talks to share? I'm about to set one up for our own little cluster (~5 machines, ~75 containers). I know tons about docker engine, and a fair bit about the registry, but it's always nice to watch a "lessons learned from actually doing this in production" talk to know what mistakes to avoid

I run tens of thousands of docker images in production, or rather, tens of thousands of copies of a few hundred images. If you do something like this, you absolutely MUST have a local registry. Harbor [1], JFrog [2], and Quay [3] would be the first ones that I look at. Harbor is open source, free, and a member of the CNCF. You will need to do a little bit of work to set it up to scale properly. JFrog offers a SaaS re…

Thank you very much. This is exactly the type of info I needed.

Re: Docker to rate limit image pulls

#264

Post by manquer »

threeseed wrote: post #225
manquer wrote: post #213

Earlier quoted context omitted.

Just to add all the major cloud service providers provide registries ACR /ECR/GCR etc . If you run k8s service with one of the them in my experience it is best to use the corresponding registry. I have pulled and run 20k times a 1GB image in less than 10-15 minutes without breaking a sweat. Finally GitHub packages offers a registry out of the box . It is great for CI and devs to access . I generally have the tags mir…

Github Docker Registry is a mess and should be avoided at all costs. 1) It is broken and unusable on Kubernetes and Docker Swarm. 2) It is flaky often returning 500 type errors. 3) It is expensive as the amount of pull bandwidth is very limited.

Github packages works with Github CI out of the box, it makes development lot easier, like I mentioned for best networking in prod you should always use the registry from your k8s Provider, mirroring the Github registry to ECR/GCR/ACR is fairly straightforward. Bandwidth costs are eliminated, network is lot more reliable intra DC.

Re: Docker to rate limit image pulls

#265

Post by heavenlyblue »

Spivak wrote: post #166
heavenlyblue wrote: post #74

Earlier quoted context omitted.

All reasonable orgs should have had their private docker repo a long time ago. Everybody else is living the pipe dream where they have externalised their risk and probably deserve the Docker treatment.

Yes but Docker achieved their goal of making it annoying as hell to not user DockerHub. Because you can run your own private repo just fine but what you want is a transparent proxy (like apt-catcher) that will let you pretend you’re using DH but actually pulling from either the cache or your private repos. All the pieces are there with private repos and “pullthrough” proxies they’re just not well integrated, seemingl…

> annoying as hell to not user DockerHub

Why? All your need to do is to use your domain name when referencing the image.

Re: Docker to rate limit image pulls

#266

Post by roleone »

Aeolos wrote: post #254
swuecho wrote: post #242

Earlier quoted context omitted.

no good document on it and it is not very important for me ( I run it on homelab). still wonder how to do it in minutes.

I use this (in a docker image) to generate certificates automatically: https://github.com/adferrand/dnsrobocert Expect to spend 1-2 hours first time you try it until you can setup the correct DNS records, API keys and configuration. Afterwards it's pretty hands off, every three months you'll receive an email from letsencrypt and you'll have to rerun this script to regenerate your certificates. Takes 2-3 minutes max (…

If you run traefik it's even easier: https://docs.traefik.io/https/acme/

Re: Docker to rate limit image pulls

#267

Post by drzaiusx11 »

tuananh wrote: post #259
drzaiusx11 wrote: post #149

Earlier quoted context omitted.

I just use ECR[1] which in many cases costs less and is fully locked down behind my AWS VPC With ECR you pay for image storage: $0.09 per GB after the first 1 GB which is free [1] https://aws.amazon.com/ecr/

are you gonna rebuild all the images that you use and push to ECR?

nope. you don't have to rebuild images to push to different registries

pull from docker hub once, push to ECR. then pull from ECR as much as wish

Re: Docker to rate limit image pulls

#268

Post by eblanshey »

terom wrote: post #231
eblanshey wrote: post #104

Earlier quoted context omitted.

Super easy to run a docker cache proxy: docker run -d -p 6000:5000 \ -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \ --restart always \ --name registry registry:2 That's it. Now fetch docker images from the IP that command is running on. Taken from gitlab: https://docs.gitlab.com/runner/install/registry_and_cache_se...

Is that going to actually help with the manifest-based rate limits? It sounds like it only caches the layers, the manifest metadata for a tag is not cached. https://docs.docker.com/registry/recipes/mirror/#what-if-the... > When a pull is attempted with a tag, the Registry checks the remote to ensure if it has the latest version of the requested content. Otherwise, it fetches and caches the latest content.

Hm you're right. I wonder if there's a way to cache a tag's metadata for a while...

Re: Docker to rate limit image pulls

#269

Post by eblanshey »

pbalau wrote: post #151
eblanshey wrote: post #106

Earlier quoted context omitted.

I was wondering how Docker-in-Docker works, but I couldn't find it dockermented anywhere. If it's using the host's Docker daemon, why do you need to mount the docker socket?

> If it's using the host's Docker daemon, why do you need to mount the docker socket? There are 2 components for docker: the daemon and the tool used to send commands to the daemon. In order for said tool to be able to send commands to the daemon, it needs a way to communicate with the daemon. Mounting the socket in the container is the easiest method. I have a "tooling" image that consists of a set of scripts (pytho…

Very interesting, thanks for sharing! I found a good article about it: https://jpetazzo.github.io/2015/09/03/do-not-use-docker-in-d...

At the end he describes mounting the socket. The tooling image which has all the dependencies needed to build will also have the docker cli installed, which is what I'm assuming you are doing.

I might just use this. Cheers!

Re: Docker to rate limit image pulls

#270

Post by LongLiveCHIEF »

jpambrun wrote: post #179

I see a lot of people mentioning the low cost, saying that it's no big deal. It's not the cost that I find annoying; it's dealing with credentials and secrets..

And docker hub account management and security management options have always been _horrible_. For example, I can't create a login token exclusive to one organization or repo on docker hub, meaning anywhere i use those credentials is a potential security risk to ALL my orgs and projects.
Post reply on HN