Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Docker builds using a remote BuildKit instance

blacksmith.sh

Post reply on HN
Print view View original

21–30 of 48 posts

Re: Faster Docker builds using a remote BuildKit instance

#21

Post by BonoboIO »

aayushshah15 wrote: post #6
rtpg wrote: post #5

I really am hopeful we come a bit full circle on builders and machines to "we buy one or two very expensive machines that run CI and builds". Caching in particular is just sitting there, waiting to be properly captured, instead of constantly churning on various machines. Of course, CI SaaSes implement a lot of caching on their end, but they also try to put people on the most anemic machines possible to try and captur…

> we buy one or two very expensive machines that run CI and builds This unfortunately does not work for orgs that have, say, more than 20 engineers. The core issue is that once you have a test suite large enough to have ~30 shards, you only need one engineer `git push`ing once to saturate those 1-2 expensive machines you've got sitting in the office. The CI workload is quite amenable to "serverless" when you get to a…

I can get a 48 core/96 threads dedicated server for 200€ a month on hetzner. The cheapest EC2 with that comes close costs 2€ per hour. I can get nearly 10 hetzner servers that run consistently for that price.

Obviously the dedicated machines are not rentable per hour, but the cloud is so much more expensive.

Re: Faster Docker builds using a remote BuildKit instance

#22

Post by aayushshah15 »

BonoboIO wrote: post #21
aayushshah15 wrote: post #6

Earlier quoted context omitted.

> we buy one or two very expensive machines that run CI and builds This unfortunately does not work for orgs that have, say, more than 20 engineers. The core issue is that once you have a test suite large enough to have ~30 shards, you only need one engineer `git push`ing once to saturate those 1-2 expensive machines you've got sitting in the office. The CI workload is quite amenable to "serverless" when you get to a…

I can get a 48 core/96 threads dedicated server for 200€ a month on hetzner. The cheapest EC2 with that comes close costs 2€ per hour. I can get nearly 10 hetzner servers that run consistently for that price. Obviously the dedicated machines are not rentable per hour, but the cloud is so much more expensive.

Well, the point was that if 4 concurrent `git push`es saturates up to 1500 vCPUs then you'd need 16 of those hetzner dedicated servers (which you have to manage the uptime for) that you're paying for for the entire month. ~4 pushes is a very small amount and an org with a few dozen or so engineers will regularly see peaks higher than this.

Additionally, you'd have to ensure some isolation across your test runs (either by running the test suites in ephemeral containers, or VMs) which is additional engineering work for something that isn't business critical.

Re: Faster Docker builds using a remote BuildKit instance

#24

Post by xyst »

> At Blacksmith, we regularly see our customer’s Docker builds taking 30 minutes or more

What’s the most common cause of builds taking this long in the first place…

Worst I have ever had was 5 minutes, but subsequent builds were reduced to under a minute due to build cache, creating multi-stage builds, and keeping the layers thin and optimizing the .dockerignore

Re: Faster Docker builds using a remote BuildKit instance

#26

Post by kapilvt »

xyst wrote: post #24

> At Blacksmith, we regularly see our customer’s Docker builds taking 30 minutes or more What’s the most common cause of builds taking this long in the first place… Worst I have ever had was 5 minutes, but subsequent builds were reduced to under a minute due to build cache, creating multi-stage builds, and keeping the layers thin and optimizing the .dockerignore

Multiarch via qemu

Re: Faster Docker builds using a remote BuildKit instance

#27

Post by astroalex »

TechSquidTV wrote: post #16

This is fairly similar in concept to what we do over at depot.dev https://depot.dev/blog/depot-magic-explained We've found that BuildKit has several inefficiencies preventing it from being as fast as it could be in the cloud, especially when dealing with simultaneous builds (common in CI). That led us to create our own optimized fork of BuildKit. The number of fine-tuning knobs you can turn running a self-hosted Buil…

Love what you're doing at Depot, keep up the good work!

Re: Faster Docker builds using a remote BuildKit instance

#28

Post by pxc »

I'm in the process of rolling out something analogous at work, where Nix jobs run inside rootless Podman containers but the Nix store and Nix daemon socket are passed through from the host, so the jobs' dependencies all persist, dependencies shared between projects are stored only once, when two concurrent jobs ask for the same dependency they both just wait for the dependency to be fetched once, etc.

We also currently have some jobs that build OCI images via the Docker/Podman CLI amd build using traditional Dockerfile/Containerfile scripts. For now those are centralized and run on just one host, on bare metal. I'd like to get those working via rootless Docker-in-Docker/Podman-in-Podman, but one thing that will be a little annoying with that is that we won't have any persistent caching at the Docker/Podman layer anymore. I suppose we'll end up using something like what's in the article to get that cache persistence back.

Re: Faster Docker builds using a remote BuildKit instance

#29

Post by adityamaru »

AkihiroSuda wrote: post #23

> endpoint: tcp://${{ secrets.BUILDKIT_HOST }}:9999 This should be protected with mTLS ( https://docs.docker.com/build/drivers/remote/ ) or SSH (`endpoint: ssh://user@host`) to avoid potential cryptomining attack, etc.

indeed, thats a good callout. We'll add this to our README over at https://github.com/useblacksmith/remote-buildkit-terraform

Re: Faster Docker builds using a remote BuildKit instance

#30

Post by adityamaru »

kapilvt wrote: post #26
xyst wrote: post #24

> At Blacksmith, we regularly see our customer’s Docker builds taking 30 minutes or more What’s the most common cause of builds taking this long in the first place… Worst I have ever had was 5 minutes, but subsequent builds were reduced to under a minute due to build cache, creating multi-stage builds, and keeping the layers thin and optimizing the .dockerignore

Multiarch via qemu

This is a common cause yeah but becoming less of an issue with increasing support for ARM runners.
Post reply on HN