Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well download the images too if there was support for it.
Docker to rate limit image pulls
101–110 of 274 posts
Re: Docker to rate limit image pulls
#102Earlier quoted context omitted.
This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again. Unless you’re using something like AWS CodeBuild that spins up a Linux/Windows container for your build environment, executes bash commands in a yaml file, and then terminates it when it is done. Nothing is stored locally after the build is fini…
What is a solution for the scenarios you have described? Amazon has ECR but it doesn’t support signing and doesn’t function as a proxy so you would miss upstream changes unless someone pushed them. Anything self hosted that supplies that functionality?
Do you really need all your private images to be derived directly from the upstream? Don't you start every image with:
FROM foo
RUN apt-get upgrade etc
Then why not have a set of base images, derived directly from upstream that get built every so often and have your private images be derived from that? This will not only relieve the stres on DockerHub and prevent you from having to pay the 5/month, but also give your security people a hook to run their tests and make your private images build faster, since all the system updates won't happen every time you change the code.
Re: Docker to rate limit image pulls
#103I originally came here to ask how folks use that many docker images in six hours (I'm mostly a Docker n00b, and not being facetious); however, after reading the article, I clicked to see how much unlimited is and it's $5 a month. Now my question has changed to: is $5 a month really a deal breaker for folks to get unlimited? Or what are the use cases where the cost is prohibitive? Open source or community projects? In…
I don’t think anyone who’s adopted systemd can call themselves a curmudgeon :)
Re: Docker to rate limit image pulls
#104Earlier quoted context omitted.
> While I agree that this is the way it's supposed to work, I have unfortunately worked at companies with "stateless" build/CI servers that download the Docker image each build. Couldn't they remain stateless but be redirected through a caching proxy? Memoization is not contrary to statelessness.
Sure, now they have to build a proxy...
Super easy to run a docker cache proxy:
docker run -d -p 6000:5000 \
-e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
--restart always \
--name registry registry:2
Re: Docker to rate limit image pulls
#105I originally came here to ask how folks use that many docker images in six hours (I'm mostly a Docker n00b, and not being facetious); however, after reading the article, I clicked to see how much unlimited is and it's $5 a month. Now my question has changed to: is $5 a month really a deal breaker for folks to get unlimited? Or what are the use cases where the cost is prohibitive? Open source or community projects? In…
It may be general resistance to a concept. More and more services are becoming subscription based, so 5$ here, 5$ there and the cost creeps further.
Re: Docker to rate limit image pulls
#106Earlier quoted context omitted.
Surprised to see Docker-in-Docker mentioned so deeply down here. It’s an extremely valid way of doing things, and non-trivial to implement a caching layer for.
Isn't Docker-in-Docker actually using the host's Docker daemon? I am mounting the docker socket in all my Docker-in-Docker containers, thus all the build tasks running on the same host can share the caches. I guess one could have docker containers that actually run docker, but I don't see a reason to do that...
I was wondering how Docker-in-Docker works, but I couldn't find it dockermented anywhere. If it's using the host's Docker daemon, why do you need to mount the docker socket?
Re: Docker to rate limit image pulls
#107I originally came here to ask how folks use that many docker images in six hours (I'm mostly a Docker n00b, and not being facetious); however, after reading the article, I clicked to see how much unlimited is and it's $5 a month. Now my question has changed to: is $5 a month really a deal breaker for folks to get unlimited? Or what are the use cases where the cost is prohibitive? Open source or community projects? In…
It’s not common to need that many pulls nor is it hard to build your own images.
If you’re deploying to a cluster with 200 machines, you could easily hit this if you use the public registry though. However, if you’re managing that size cluster you can probably afford the fee, but more importantly, you should probably pull once to a local registry and use that to deploy to your cluster anyway.
Re: Docker to rate limit image pulls
#108Earlier quoted context omitted.
> This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again. You're assuming that the set of build nodes is relatively static. Plenty of architectures set up autoscaling for the underlying nodes, that terminate servers that aren't being used and relatively soon enough (tens of minutes, hours) spin up ne…
If you're doing this, you're in a cloud environment that's also proximate to a blob store, and can trivially host your own registry.
> and can trivially host your own registry
That is far from trivial.
Re: Docker to rate limit image pulls
#109I originally came here to ask how folks use that many docker images in six hours (I'm mostly a Docker n00b, and not being facetious); however, after reading the article, I clicked to see how much unlimited is and it's $5 a month. Now my question has changed to: is $5 a month really a deal breaker for folks to get unlimited? Or what are the use cases where the cost is prohibitive? Open source or community projects? In…
$5/month isn't $5/month. It's convincing your boss you need $5/month, because they need to convince their boss, which eventually makes its way up the chain to C-levels, who don't know Docker from yesterday's rotting tuna casserole and view eating either that or the $5/month with the same level of disdain.
It isn't about the money, it's about the Mommy-May-I up and down the chain with emails and meetings and careful explanations to skeptical glares. It's a psychological and institutional barrier.
Re: Docker to rate limit image pulls
#110I originally came here to ask how folks use that many docker images in six hours (I'm mostly a Docker n00b, and not being facetious); however, after reading the article, I clicked to see how much unlimited is and it's $5 a month. Now my question has changed to: is $5 a month really a deal breaker for folks to get unlimited? Or what are the use cases where the cost is prohibitive? Open source or community projects? In…
It’s not common to need that many pulls nor is it hard to build your own images. If you’re deploying to a cluster with 200 machines, you could easily hit this if you use the public registry though. However, if you’re managing that size cluster you can probably afford the fee, but more importantly, you should probably pull once to a local registry and use that to deploy to your cluster anyway.
Do you run a local registry? Any high-quality articles/youtube talks to share? I'm about to set one up for our own little cluster (~5 machines, ~75 containers). I know tons about docker engine, and a fair bit about the registry, but it's always nice to watch a "lessons learned from actually doing this in production" talk to know what mistakes to avoid