Live data from Hacker News

Docker to rate limit image pulls

docker.com

251–260 of 274 posts

Re: Docker to rate limit image pulls

#251

Earlier quoted context omitted.

Do you run a local registry? Any high-quality articles/youtube talks to share? I'm about to set one up for our own little cluster (~5 machines, ~75 containers). I know tons about docker engine, and a fair bit about the registry, but it's always nice to watch a "lessons learned from actually doing this in production" talk to know what mistakes to avoid

Just drop a Sonatype Nexus instance on a Docker container somewhere on your network. Alternatively, use Squid if you don't push to the public Docker registry, although you might need to mess around with internal CA for SSL...

Docker supports proxies (they call them “pullthrough repos”) so you don’t have to be so generic as an http proxy.

Re: Docker to rate limit image pulls

#252

Earlier quoted context omitted.

> the current culture regarding builds and CI in general is horrifyingly ineffficient I used to be in charge of the website for a company you’ve heard of. We once realized some huge proportion of our traffic originated from a hosted CI company requesting the site thousands and thousands of times (guessing one for each build they hosted) every 5 minutes. I can’t remember what proportion of traffic it was but I’m prett…

I sure hope the "cloud first" advocates are happy now, because they have managed to create masses of developers who have next to no idea that what they're doing is an incredible waste of resources. These are also the same people who are perplexed why their systems intermittently fail, or are surprised that they do when the Internet connection cuts out for a bit.

We just emailed them and asked them to stop. Which they did, and they apologized, but maybe they just changed it to google.com or something which wouldn’t be any less wasteful.

Re: Docker to rate limit image pulls

#253

Earlier quoted context omitted.

Just drop a Sonatype Nexus instance on a Docker container somewhere on your network. Alternatively, use Squid if you don't push to the public Docker registry, although you might need to mess around with internal CA for SSL...

Nexus in a container... because storage in containers is such a good idea? Any vps with a disk is probably a better idea

Storage in containers has been a long solved issue. The defaults are unfortunate because but make sense for ease of use. Your container root should be read only, ephemeral storage lives in a tmpfs or dynamic volumes depending on performance and size needs, and persistent storage lives in volumes.

Re: Docker to rate limit image pulls

#254

Earlier quoted context omitted.

If you cannot get a TLS cert for internal infrastructure in a few minutes, I'd recommend you start looking into why.

no good document on it and it is not very important for me ( I run it on homelab). still wonder how to do it in minutes.

I use this (in a docker image) to generate certificates automatically: https://github.com/adferrand/dnsrobocert

Expect to spend 1-2 hours first time you try it until you can setup the correct DNS records, API keys and configuration.

Afterwards it's pretty hands off, every three months you'll receive an email from letsencrypt and you'll have to rerun this script to regenerate your certificates. Takes 2-3 minutes max (but of course you still need to distribute your certificates to all relevant services...)

Re: Docker to rate limit image pulls

#255
Hrm, based on this: https://docs.docker.com/docker-hub/orgs/#add-a-member-to-a-t...

I see that Docker doesn't actually offer an AWS-style enterprise account that one can use to hand authorization to developers without requiring those developers to make individual accounts.

It feels pretty sassy of docker to give everyone 2 months to shove credentials everywhere when docker themselves haven't done the minimum to make enterprise accounts realistic. Instead, they're adopting the github model of "oh, just ask everyone to make personal accounts and then include their personal accounts in the org team". That has problems.

Firstly, it puts employers in the unpleasant position of attempting to compel employees to make legal agreements with third parties (docker, in this case). The correct way to do this is AWS-style, where the org itself makes /one/ agreement and then delegates that agreement via access keys. This is the minimum I expect from enterprise account systems, hard fail for docker.

Secondly, it's a clusterfuck to manage. You end up with an org filled with random-arse account names that you can't really audit, and you don't know who has access to what. If employees leave the org, it's hard to ensure that their access is revoked because the access takes place entirely outside the standard account domains.

Github has recently improved this a shade by adding ADFS authorization to org accounts, but that involves asking employees to tie their personal (and all github and docker accounts /are/ personal) account to their work ADFS account, which is a shitty half-solution.

All things considered, docker made this problem for themselves. They've spent /years/ working hard to get everyone to make docker accounts and push everything to docker hub instead of fostering an ecosystem of registries by different orgs for different purposes. All of a sudden it's now "too expensive" and they're dropping the hammer on everyone to sign up and push credentials everywhere with very little warning, whilst not doing their half of the work by making a proper delegated authority account system.

Doesn't fill me with confidence for their future as a stable platform on which to base a business.

Re: Docker to rate limit image pulls

#256
post #246

Earlier quoted context omitted.

"Protesters say their peaceful assembly has been disrupted and escalated by the police. The police argue they've only been intervening in riots to maintain order and protect property." Done.

If you have to represent "both sides" (in many cases there'll be more than two sides really), you end up having to give a voice to nutjobs, plus you present both sides as equally valid assessments. Much as we'd all love an "unbiased" news source, the reality is that bias is a very hard problem to solve well.

Indeed, just repeating what people say about an event may be factual, but without any concept of what is actually true, it can’t be considered objective.

If one side is lying, objective reporting would tell you which side it was.

Re: Docker to rate limit image pulls

#257
post #67

Earlier quoted context omitted.

Artifactory is less bad than most of the tools I have to use all day.

Artifactory is the very definition of expensive (even at an enterprise scale) when it comes to docker images though.

Can you tell me more? How expensive are we talking?

Working for the same sized companies for a while has apparently dulled my senses. At a certain size, the capital that matters is the political capital it takes to get a vendor agreement in place to begin with. The monthly costs of the system are something you only feel through pushback on how big the repo gets, or the rate of traffic (experiencing the latter now with a browser testing SaaS)

Re: Docker to rate limit image pulls

#258

Earlier quoted context omitted.

Do you run a local registry? Any high-quality articles/youtube talks to share? I'm about to set one up for our own little cluster (~5 machines, ~75 containers). I know tons about docker engine, and a fair bit about the registry, but it's always nice to watch a "lessons learned from actually doing this in production" talk to know what mistakes to avoid

I run tens of thousands of docker images in production, or rather, tens of thousands of copies of a few hundred images. If you do something like this, you absolutely MUST have a local registry. Harbor [1], JFrog [2], and Quay [3] would be the first ones that I look at. Harbor is open source, free, and a member of the CNCF. You will need to do a little bit of work to set it up to scale properly. JFrog offers a SaaS re…

We have not had good luck with Quay. They are not stable, especially as of late. There was a period last month where for two weeks pulling images was a crapshoot.

Re: Docker to rate limit image pulls

#259

Earlier quoted context omitted.

Do you run a local registry? Any high-quality articles/youtube talks to share? I'm about to set one up for our own little cluster (~5 machines, ~75 containers). I know tons about docker engine, and a fair bit about the registry, but it's always nice to watch a "lessons learned from actually doing this in production" talk to know what mistakes to avoid

I just use ECR[1] which in many cases costs less and is fully locked down behind my AWS VPC With ECR you pay for image storage: $0.09 per GB after the first 1 GB which is free [1] https://aws.amazon.com/ecr/

are you gonna rebuild all the images that you use and push to ECR?

Re: Docker to rate limit image pulls

#260

Good! I'd even welcome much more agressive limits than what they're proposing; the current culture regarding builds and CI in general is horrifyingly ineffficient, wasteful and in the end just plain slow. I'm looking forward to developers adjusting their workflows (and caches, etc.) to actual, reasonable limits, not just perusing the service as if it were an unlimited cost-free cornucopia of software.

Everything is inefficient. There is a huge class of "developers" that don't understand what O(n) means, and a subset of them are vocally proud of it.
Post reply on HN