Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Docker to rate limit image pulls

docker.com

Post reply on HN
Print view View original

21–30 of 274 posts

Re: Docker to rate limit image pulls

#21

Post by pm90 »

kzrdude wrote: post #13
uberman wrote: post #3

I'm very conflicted about this. On the one hand, I recognize that there are potentially significant costs to be born to serve these repositories. On the other hand, making docker part of your infrastructure requires a certain degree of availability. At some level this seems to me like using my IDE and after 6 hours it would stop working or finding that my CDNJS references to bootstrap stopped working after 6 hours of…

It really sounds like a problem we just have to solve. Putting all the expectation on one central service is not reliable.. it should be a distributed network of content delivery nodes, of which you run a few yourself.

I like this idea. I would think that the “default” hub should not be managed only by Docker, but eg by docker and other registries. I suspect that docker inc still wants full control over their registry though so I don’t see this happening.

We might decide on another, non dockerhub open registry though, and use that instead.

Re: Docker to rate limit image pulls

#22

Post by mikepurvis »

throwanem wrote: post #12
uberman wrote: post #3

I'm very conflicted about this. On the one hand, I recognize that there are potentially significant costs to be born to serve these repositories. On the other hand, making docker part of your infrastructure requires a certain degree of availability. At some level this seems to me like using my IDE and after 6 hours it would stop working or finding that my CDNJS references to bootstrap stopped working after 6 hours of…

Do pulls for a "latest" tag bypass the cache?

No they don't, but if you do something silly like spin up a bare VM build node, and then pull your environment every time, then obviously you won't get any caching.

Re: Docker to rate limit image pulls

#23

Post by verdverm »

throwanem wrote: post #12
uberman wrote: post #3

I'm very conflicted about this. On the one hand, I recognize that there are potentially significant costs to be born to serve these repositories. On the other hand, making docker part of your infrastructure requires a certain degree of availability. At some level this seems to me like using my IDE and after 6 hours it would stop working or finding that my CDNJS references to bootstrap stopped working after 6 hours of…

Do pulls for a "latest" tag bypass the cache?

No, latest is just a tag and is not handled differently. There is a CLI flag to ignore cached layers. This however does not effect the FROM line, so if you have the image, there will be no pull.

Expect this to effect CI systems

Re: Docker to rate limit image pulls

#24

Post by saagarjha »

Crazyontap wrote: post #19

Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well download the images too if there was support for it.

> Most of us download our OS via torrents only

I mean, I only do it to stick it to the people who claim torrents can only be used for piracy; I think most people prefer the simplicity of direct downloads though…

Re: Docker to rate limit image pulls

#25

Post by bsamuels »

Crazyontap wrote: post #19

Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well download the images too if there was support for it.

The hardest part of that would be verifying image authenticity.

Google Cloud uses an adjacent feature called binary authorization. When turned on, only images that are signed by a given authority (usually your ci/cd instruments) can be run inside your Kubernetes cluster.

Binary authorization may be a good starting point for someone trying to make bittorrent distributed images a usable thing.

Re: Docker to rate limit image pulls

#27

Post by sneak »

bsamuels wrote: post #25
Crazyontap wrote: post #19

Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well download the images too if there was support for it.

The hardest part of that would be verifying image authenticity. Google Cloud uses an adjacent feature called binary authorization. When turned on, only images that are signed by a given authority (usually your ci/cd instruments) can be run inside your Kubernetes cluster. Binary authorization may be a good starting point for someone trying to make bittorrent distributed images a usable thing.

> The hardest part of that would be verifying image authenticity.

That's exactly what Bittorrent does with its hash tree. You'd get the root hash (extremely tiny) from Docker Hub, and the rest of the metadata, as well as the data blocks, from the swarm. The authenticity is all handled by the TLS that serves you the root infohash from Docker Hub. It's a Merkle tree: the root hash is for the metadata, which is a list of hashes of the blocks.

Re: Docker to rate limit image pulls

#28

Post by pm90 »

Keverw wrote: post #5

I kinda wonder if Docker as a company is struggling. Redhat made Podman which is a compatible replacement, Then there's swarm but apparently that's not recommended and actively developed anymore, then as far as I know they sold off their enterprise clustering product. Seems Kubernetes is the popular thing now even if a bit complex to setup. Wonder what the current business model? Pretty neat idea of using containers,…

They won the container war but lost the orchestration war. Even if docker compose was successful though, I fail to see how the clouds wouldn’t just replicate everything. So I guess they just failed to monetize the technology.

Re: Docker to rate limit image pulls

#29

Post by scarface74 »

bsamuels wrote: post #14

There is very little reason for a build node to need to pull 200 images in 6 hours, and here is why: When a machine issues a ``docker build`` command, the program reads the relevant dockerfile to check for any base images that need to be pulled (a la "FROM:") These base images are identified based on the image repository, image name, and image tag. The first thing docker does is it checks its local registry and tries…

This is significant - if your organization only uses a few dozen base images from DockerHub, those images will only be downloaded by each build node _once_, then never again.

Unless you’re using something like AWS CodeBuild that spins up a Linux/Windows container for your build environment, executes bash commands in a yaml file, and then terminates it when it is done. Nothing is stored locally after the build is finished.

I’m sure there are other similar services. Wouldn’t Azure Devops using hosted builds do basically the same thing? I haven’t used it since they changed the name from Visual Studio Team Services.

Re: Docker to rate limit image pulls

#30

Post by pixl97 »

bsamuels wrote: post #25
Crazyontap wrote: post #19

Honest question, but why can't docker use something like bittorrent to download images? Most of us download our OS via torrents only, so we may as well download the images too if there was support for it.

The hardest part of that would be verifying image authenticity. Google Cloud uses an adjacent feature called binary authorization. When turned on, only images that are signed by a given authority (usually your ci/cd instruments) can be run inside your Kubernetes cluster. Binary authorization may be a good starting point for someone trying to make bittorrent distributed images a usable thing.

Why, the website could still have the hash for the image which is only a few KB versus hundreds of megabytes or even gigabytes. Just have the bit-docker app check the hash before executing.
Post reply on HN