Live data from Hacker News

Running our Docker registry on-prem with Harbor

dev.37signals.com

41–50 of 78 posts

Re: Running our Docker registry on-prem with Harbor

#41
> pulling and pushing our images over the internet dozens of times a day caused us to hit the contracted bandwidth limit with our datacenter provider Deft repeatedly

I wonder what they were doing that resulted in blowing out their Docker layer cache on every pull and push.

Normally only a layer diff would be sent over the wire, such as a code change that didn't change your dependencies.

Re: Running our Docker registry on-prem with Harbor

#42
post #3

Would be interested to see a cost breakdown for the ECR vs S3 and compute cost.

ECR is kind of hard to beat if you're ok with being in the cloud.

The last time I used it earlier this year for a company already on AWS, it was ~$3 / month per region to store 8 private repos and it was really painless to have a flexible and automated life cycle policy that deleted old image tags. It supports cross region replication too. All of that comes without maintenance or compute costs and if you're already familiar with Terraform, etc. you can automate all of that in a few hours of dev time.

Re: Running our Docker registry on-prem with Harbor

#43
post #13

Earlier quoted context omitted.

Do they? If all this is in their DC then it’s going over their wires, unless their colo provider somehow had visibility into their traffic (which I’d guess they don’t).

It's in Deft's ORD and IAD data centers, using their network for ingress/egress. Still has to go over transit between those two locations.

But that might be baked into their enterprise pricing. Since it’s still “within” Deft. Site to site is common.

Re: Running our Docker registry on-prem with Harbor

#44
So what are the thoughts of folks who have used Nexus and moved to Harbor?

In my experience Nexus is a bit weird to administer and sometimes the Docker cleanup policies straight up don't work (junk left over in the blob stores even if you try to clean everything), but it also supports all sorts of other formats, such as repositories for Maven and NuGet. Kind of hungry in regards to resources, though.

Re: Running our Docker registry on-prem with Harbor

#45
I am not to familiar with Kamal but it seems possible to integrate it with my project Spegel to remove some of the load from upstream. Especially if they are running clusters of servers physically located close to each other they could avoid some of the replication complexity with multiple Harbor instances.

Re: Running our Docker registry on-prem with Harbor

#46

So what are the thoughts of folks who have used Nexus and moved to Harbor? In my experience Nexus is a bit weird to administer and sometimes the Docker cleanup policies straight up don't work (junk left over in the blob stores even if you try to clean everything), but it also supports all sorts of other formats, such as repositories for Maven and NuGet. Kind of hungry in regards to resources, though.

We run both Nexus and Harbor. I am about to dump Harbor because teams don’t use it and frankly Nexus provides the same functionality.

Re: Running our Docker registry on-prem with Harbor

#47

Why does the Harbor VM need 32 cores and 64GB of RAM? Especially if it's only serving 32,000 pulls over 2 months.

> Why does the Harbor VM need ...

They never say that it needs these resources, they say that the current VM has this config. Probably overkill by 1000%.

Re: Running our Docker registry on-prem with Harbor

#48

So what are the thoughts of folks who have used Nexus and moved to Harbor? In my experience Nexus is a bit weird to administer and sometimes the Docker cleanup policies straight up don't work (junk left over in the blob stores even if you try to clean everything), but it also supports all sorts of other formats, such as repositories for Maven and NuGet. Kind of hungry in regards to resources, though.

Nexus can be flaky, but it's pretty universal as you say. Harbor is a hard sell for me, since generally in any organization you'll need non-OCI artifact storage at some point, and maintaining 2 tools is always a pain.

Re: Running our Docker registry on-prem with Harbor

#50

Why does the Harbor VM need 32 cores and 64GB of RAM? Especially if it's only serving 32,000 pulls over 2 months.

For what it's worth, I have an on-prem Nexus server with a Docker repository. It has 8 cores and 16GB RAM. It has 82000 hits/day in the webserver log, though 99.9% of them transfer only a few kB, so I assume it's a metadata check and the client already has the correct version.

The same Nexus is also hosting our Maven and NodeJS repositories. That has 1,800,000 hits per day, although all but 120,000 of them return HTTP 404. (I think one of our clients has misconfigured their build server to use our Maven repository as a mirror of Maven Central, but as it's just an AWS IP I don't know who it is.)

I'm sure it's overprovisioned, but the marginal cost of 2, 4 or 8 cores and 4, 8 or 16GB RAM isn't much when we buy our own hardware.

Post reply on HN