Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running our Docker registry on-prem with Harbor

dev.37signals.com

Post reply on HN
Print view View original

61–70 of 78 posts

Re: Running our Docker registry on-prem with Harbor

#61

Post by nchmy »

Symbiote wrote: post #50
fastest963 wrote: post #4

Why does the Harbor VM need 32 cores and 64GB of RAM? Especially if it's only serving 32,000 pulls over 2 months.

For what it's worth, I have an on-prem Nexus server with a Docker repository. It has 8 cores and 16GB RAM. It has 82000 hits/day in the webserver log, though 99.9% of them transfer only a few kB, so I assume it's a metadata check and the client already has the correct version. The same Nexus is also hosting our Maven and NodeJS repositories. That has 1,800,000 hits per day, although all but 120,000 of them return HTT…

> the marginal cost of 2, 4 or 8 cores and 4, 8 or 16GB RAM isn't much when we buy our own hardware

this is the crux of it all. RENTING bare metal (eg from hetzner) is 10x cheaper than aws ec2. So, I can only imagine how much cheaper it is when you buy the hardware directly.

Re: Running our Docker registry on-prem with Harbor

#62

Post by Jedd »

phireal wrote: post #23
muhehe wrote: post #17

This looks nice. What would be good on-prem S3 companion for this? I know if minio but I think there was some recent drama about it (I don't know specifics, just a feeling)

Minio used to be de facto here, but they did a bait and switch recently and removed the UI from the free version. Garage is probably closest to best in class for open source on prem.

Does garage-dev have a GUI? I thought it did not.

I'm using it in my lab, along with an older instance of minio, and both are excellent choices I think. (I'm running both as HA jobs within a Hashicorp Nomad cluster, which complicates / eases various things.)

I had a vague memory of minio losing some favour a while back because they switched their underlying storage from basic object : normal file mapping to something more complex. A lot of home users, and I guess developers, liked the ability to navigate their S3-esque buckets via normal linux file system tooling.

Re: Running our Docker registry on-prem with Harbor

#63

Post by Jedd »

muhehe wrote: post #17

This looks nice. What would be good on-prem S3 companion for this? I know if minio but I think there was some recent drama about it (I don't know specifics, just a feeling)

Do you actually need S3-alike?

I run distribution [0] / registry [1] as a docker (via Nomad) job, and it just uses a shared nfs mount in my cluster - ie. EXT4 FS underneath.

Currently has 12GB and 52 repositories, and is very performant. It's just a file server.

Unless your size requirements are huge, the complexity of having your registry rely on a self-hosted S3-alike sounds like a lot of pain for future-muhehe to contend with if / when the S3-alike hiccups.

[0] https://github.com/distribution/distribution

[1] https://github.com/Joxit/docker-registry-ui

Re: Running our Docker registry on-prem with Harbor

#64

Post by mdaniel »

tedivm wrote: post #51
lijok wrote: post #35

We self-host Harbor as well, it’s fairly painless. Has SSO out of the box, a Terraform provider that covers everything, and for the most part just works. The issues we’ve had so far: - No programmatic way to retrieve your token that’s required for ‘docker login’. So we had to create a robot account per user and pop their creds into our secrets store. - Migrating between sites by cloning the underlying S3 bucket and s…

The lack of OIDC support for Harbor has been the biggest annoyance for me. I'd love to be able to push from Github Actions to Harbor without needing robot users.

I was shocked to read such a thing in 2025 but either there is some nuance to your observation or your information is outdated https://goharbor.io/docs/2.13.0/administration/#:~:text=or%2...

Re: Running our Docker registry on-prem with Harbor

#65

Post by mdaniel »

redblueflame wrote: post #20
muhehe wrote: post #17

This looks nice. What would be good on-prem S3 companion for this? I know if minio but I think there was some recent drama about it (I don't know specifics, just a feeling)

If you want to have replication built in, you can give https://garagehq.deuxfleurs.fr/ a try

Just make sure the AGPLv3 is compatible with your policies https://git.deuxfleurs.fr/Deuxfleurs/garage/src/tag/v2.0.0/L...

Re: Running our Docker registry on-prem with Harbor

#66

Post by mdaniel »

arccy wrote: post #34
muhehe wrote: post #17

This looks nice. What would be good on-prem S3 companion for this? I know if minio but I think there was some recent drama about it (I don't know specifics, just a feeling)

https://github.com/seaweedfs/seaweedfs much simpler than minio

https://github.com/seaweedfs/seaweedfs#compared-to-other-fil... is handy, and I like that they support "sane" metadata stores and don't just do something dumb like make me run etcd

Re: Running our Docker registry on-prem with Harbor

#67

Post by mdaniel »

inglor_cz wrote: post #52

We run our own Docker registry on-prem with Harbor as well. One issue to solve is auto-deletion of old images, so that the storage does not swell. Any tips?

https://goharbor.io/docs/2.13.0/working-with-projects/workin... seems like what you want, especially in combination with https://goharbor.io/docs/2.13.0/administration/garbage-colle...

Re: Running our Docker registry on-prem with Harbor

#68

Post by muhehe »

Jedd wrote: post #63
muhehe wrote: post #17

This looks nice. What would be good on-prem S3 companion for this? I know if minio but I think there was some recent drama about it (I don't know specifics, just a feeling)

Do you actually need S3-alike? I run distribution [0] / registry [1] as a docker (via Nomad) job, and it just uses a shared nfs mount in my cluster - ie. EXT4 FS underneath. Currently has 12GB and 52 repositories, and is very performant. It's just a file server. Unless your size requirements are huge, the complexity of having your registry rely on a self-hosted S3-alike sounds like a lot of pain for future-muhehe to…

For image registry...probably not that much. But it is quite common for other software, often those that target containers/k8s/... I already have few of those and that would be quite handy to have some good local s3.

Re: Running our Docker registry on-prem with Harbor

#69

Post by Jedd »

muhehe wrote: post #68
Jedd wrote: post #63

Earlier quoted context omitted.

Do you actually need S3-alike? I run distribution [0] / registry [1] as a docker (via Nomad) job, and it just uses a shared nfs mount in my cluster - ie. EXT4 FS underneath. Currently has 12GB and 52 repositories, and is very performant. It's just a file server. Unless your size requirements are huge, the complexity of having your registry rely on a self-hosted S3-alike sounds like a lot of pain for future-muhehe to…

For image registry...probably not that much. But it is quite common for other software, often those that target containers/k8s/... I already have few of those and that would be quite handy to have some good local s3.

Concur - as noted in a sibling comment, I'm using both on Nomad/docker 5-node clusters, but started with minio then ran up garage-dev to compare (I just haven't migrated data over from minio yet).

I use mine primarily for long-term stores for Grafana's Mimir and Loki products.

Anyway, minio & garage are both lightweight, and happily run in docker, so if you've got a small cluster at home it would take maybe an hour or two to install them both from scratch.

Re: Running our Docker registry on-prem with Harbor

#70

Post by Symbiote »

nchmy wrote: post #61
Symbiote wrote: post #50

Earlier quoted context omitted.

For what it's worth, I have an on-prem Nexus server with a Docker repository. It has 8 cores and 16GB RAM. It has 82000 hits/day in the webserver log, though 99.9% of them transfer only a few kB, so I assume it's a metadata check and the client already has the correct version. The same Nexus is also hosting our Maven and NodeJS repositories. That has 1,800,000 hits per day, although all but 120,000 of them return HTT…

> the marginal cost of 2, 4 or 8 cores and 4, 8 or 16GB RAM isn't much when we buy our own hardware this is the crux of it all. RENTING bare metal (eg from hetzner) is 10x cheaper than aws ec2. So, I can only imagine how much cheaper it is when you buy the hardware directly.

At a very rough estimate with Hetzner's calculator, which doesn't have exactly the systems I most recently bought, our servers cost about 18 months of renting the closest Hetzner dedicated server.

Add to that rackspace costs, staff time doing installation (or cost of remote hands etc). I don't have our figures for rackspace costs, but staff time is fairly low — an hour or two when the server is purchased and later thrown out, and maybe the same to replace a HDD at some point in the 5-8 years we keep it.

Also add the time to get competing quotes from Dell, HP etc and decide on the configuration, but then a similar process is needed to choose from Hetzner vs their competitors.

Post reply on HN