Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Docker builds using a remote BuildKit instance

blacksmith.sh

Post reply on HN
Print view View original

41–48 of 48 posts

Re: Faster Docker builds using a remote BuildKit instance

#41

Post by adamcharnock »

aayushshah15 wrote: post #22
BonoboIO wrote: post #21

Earlier quoted context omitted.

I can get a 48 core/96 threads dedicated server for 200€ a month on hetzner. The cheapest EC2 with that comes close costs 2€ per hour. I can get nearly 10 hetzner servers that run consistently for that price. Obviously the dedicated machines are not rentable per hour, but the cloud is so much more expensive.

Well, the point was that if 4 concurrent `git push`es saturates up to 1500 vCPUs then you'd need 16 of those hetzner dedicated servers (which you have to manage the uptime for) that you're paying for for the entire month. ~4 pushes is a very small amount and an org with a few dozen or so engineers will regularly see peaks higher than this. Additionally, you'd have to ensure some isolation across your test runs (eithe…

In my experience, dedicated hardware has provided a baseline real-world 2x speed up over cloud instances (presumably down to no contention, local nvme). So that would be 8 hetzner instances.

I managed to squeeze out a 5x speed up total (see my other comment). In which case that would mean 3-4 instances.

Plus with shorter build times you may then find that having a builds queued up is acceptable.

Re: Faster Docker builds using a remote BuildKit instance

#42

Post by suryao »

rtpg wrote: post #32
aayushshah15 wrote: post #6

Earlier quoted context omitted.

> we buy one or two very expensive machines that run CI and builds This unfortunately does not work for orgs that have, say, more than 20 engineers. The core issue is that once you have a test suite large enough to have ~30 shards, you only need one engineer `git push`ing once to saturate those 1-2 expensive machines you've got sitting in the office. The CI workload is quite amenable to "serverless" when you get to a…

Sounds like somebody should set up incremental CI with Bazel :) Seriously though, of course there's a lot of details here, but I think people tend to not really interenalize how much testing is about confidence, and things like incremental CI can really chew away at how big/small your test suite needs to be. There are some things that are just inherently slow, but I've seen a lot of test suites that are mostly rerunn…

incremental CI is absolutely the way to go

Re: Faster Docker builds using a remote BuildKit instance

#43

Post by pxc »

selcuka wrote: post #37
delduca wrote: post #2

I prefer to use a VPS

Well, EC2 is a type of VPS.

'Cloud VPS' is generally a lot more expensive than the cheapest old-school VPS providers— sometimes like 10x, similar to the numbers commenters elsewhere were discussing for dedicated servers vs. EC2.

Re: Faster Docker builds using a remote BuildKit instance

#44

Post by imtringued »

zerotolerance wrote: post #36

There are a few tragedies in the Docker story, but at least two are specifically tied to naming things. First, Swarm (mode) because by the time they released Swarm (mode) the world had already taken a collective dump on Swarm (the proof-of-concept). Even in 2024 most of the time people talk about Swarm and start dumping on it they're actually talking about the proof-of-concept architecture. Second, they should never…

I don't see how this is a tragedy. You're blowing something trivial out of proportion.

When you build alpine packages, you literally have to call abuild on your APKBUILD files. It's the same for Arch Linux. The files are called PKGBUILD. So even if you decide to package your applications (uh, using docker run? that changes nothing!) before docker build and then install them with the OS package manager, you will run into exactly the same thing.

Re: Faster Docker builds using a remote BuildKit instance

#45

Post by imtringued »

zerotolerance wrote: post #35
xyst wrote: post #24

> At Blacksmith, we regularly see our customer’s Docker builds taking 30 minutes or more What’s the most common cause of builds taking this long in the first place… Worst I have ever had was 5 minutes, but subsequent builds were reduced to under a minute due to build cache, creating multi-stage builds, and keeping the layers thin and optimizing the .dockerignore

People doing all the work of dependency fetches, code builds, and test execution inside ephemeral environments never designed for building software within.

Fetching packages isn't the problem. The problem is the lack of "out of the box" caching pf the downloaded packages. You'll have to do that yourself with artifactory and Docker does not nudge you towards doing that, at all.

Re: Faster Docker builds using a remote BuildKit instance

#46

Post by jjayj »

bhouston wrote: post #18
jjayj wrote: post #17

Earlier quoted context omitted.

I regularly build images where we install Python from source, which makes 30m seem quite normal...

Why?

We do development/builds in containers to make things easier on the devs. These are the Python build containers, so we have to rebuild every time there's a new version of Python or one of its dependencies.

Re: Faster Docker builds using a remote BuildKit instance

#48

Post by TechSquidTV »

astroalex wrote: post #27
TechSquidTV wrote: post #16

This is fairly similar in concept to what we do over at depot.dev https://depot.dev/blog/depot-magic-explained We've found that BuildKit has several inefficiencies preventing it from being as fast as it could be in the cloud, especially when dealing with simultaneous builds (common in CI). That led us to create our own optimized fork of BuildKit. The number of fine-tuning knobs you can turn running a self-hosted Buil…

Love what you're doing at Depot, keep up the good work!

Thank you!
Post reply on HN