Live data from Hacker News

Faster Docker builds using a remote BuildKit instance

blacksmith.sh

31–40 of 48 posts

Re: Faster Docker builds using a remote BuildKit instance

#31
post #28

I'm in the process of rolling out something analogous at work, where Nix jobs run inside rootless Podman containers but the Nix store and Nix daemon socket are passed through from the host, so the jobs' dependencies all persist, dependencies shared between projects are stored only once, when two concurrent jobs ask for the same dependency they both just wait for the dependency to be fetched once, etc. We also current…

> Nix jobs run inside rootless Podman containers but the Nix store and Nix daemon socket are passed through from the host

That's a neat idea, was the primary motivation for building this out the perf gains on the table?

Re: Faster Docker builds using a remote BuildKit instance

#32
post #5

I really am hopeful we come a bit full circle on builders and machines to "we buy one or two very expensive machines that run CI and builds". Caching in particular is just sitting there, waiting to be properly captured, instead of constantly churning on various machines. Of course, CI SaaSes implement a lot of caching on their end, but they also try to put people on the most anemic machines possible to try and captur…

> we buy one or two very expensive machines that run CI and builds This unfortunately does not work for orgs that have, say, more than 20 engineers. The core issue is that once you have a test suite large enough to have ~30 shards, you only need one engineer `git push`ing once to saturate those 1-2 expensive machines you've got sitting in the office. The CI workload is quite amenable to "serverless" when you get to a…

Sounds like somebody should set up incremental CI with Bazel :)

Seriously though, of course there's a lot of details here, but I think people tend to not really interenalize how much testing is about confidence, and things like incremental CI can really chew away at how big/small your test suite needs to be. There are some things that are just inherently slow, but I've seen a lot of test suites that are mostly rerunning tests that only use unchanged code for most of its runtime.

My glib assertion is that there is likely to be no test suite generated by 20 engineers that requires 30 shards that is impossible to chop up with incremental CI. And downstream of that, getting incremental CI would improve DX a lot, cuz I bet those 30 shards take a long time

Re: Faster Docker builds using a remote BuildKit instance

#33
post #28

I'm in the process of rolling out something analogous at work, where Nix jobs run inside rootless Podman containers but the Nix store and Nix daemon socket are passed through from the host, so the jobs' dependencies all persist, dependencies shared between projects are stored only once, when two concurrent jobs ask for the same dependency they both just wait for the dependency to be fetched once, etc. We also current…

> Nix jobs run inside rootless Podman containers but the Nix store and Nix daemon socket are passed through from the host That's a neat idea, was the primary motivation for building this out the perf gains on the table?

More or less! Before that we basically had two kinds of jobs: one where Docker provided the job's dependencies and one where Nix provided the job's dependencies. The former was, naturally, always containerized, but the latter ran on bare metal. (We also had some jobs that used Nix without a persistent /nix/store, but never mind that.) Given both options, choosing Nix over Docker is really nice in two ways: (1) you don't need to deal with any infrastructure (i.e., push an image somewhere) for your job; and (2) even so, the second run and later can be really, really fast. We also use Nix to provide dependencies on our local machines for some projects, so just reusing that same environment in CI is a natural fit, too.

But as we started to mature our own CI 'infrastructure' (the automation we use to set up our self-hosted runners), I wanted to containerize the Nix builds. Using 'shell executors' in GitLab just feels icky to me, like a step backwards into Jenkins hell. Those jobs do leave a little bit more behind on disk. More importantly, though, while all of my team's Nix jobs use Nix in an ephemeral way, it is possible to run `nix profile install ...` in one of these bare metal jobs. That could potentially affect other such jobs, plus it creates a 'garbage collector root' that will reduce how much `nix-collect-garbage` can clean up a little bit. Our jobs are ones we'd like other teams across the company to run, and so we also want to provide some really low-effort ways for them to do so, namely: via shared infrastructure we host, via any Docker-capable runners they might already have, and by leveraging the same IaC we use to stand up our own runners.

To that end, we really want to have just one type of job that requires just one type of execution environment, and we definitely want opt-in persistence instead of a mess where jobs can very easily influence one another by accident or malice. But we don't want to lose the speedup! The real action in these jobs is small, so by sharing a persistent Nix store between runs, they go down from 2-10 minutes to 2-10 seconds, which is the kind of UX we want for our internal customers.

The new Nix image is more suitable for all three target scenarios: it's less risky on runner hosts shared by multiple teams, it still works normally (downloading deps via Nix on every run) on 'naive' Docker/Podman setups, and our runner initialization script actually uses Nix to provide Docker and Podman (both rootless), so any team can use it on top of whatever VM images they're already using for their CI runners regardless of distro or version once they're ready to opt into that performance optimization.

Re: Faster Docker builds using a remote BuildKit instance

#35
post #24

> At Blacksmith, we regularly see our customer’s Docker builds taking 30 minutes or more What’s the most common cause of builds taking this long in the first place… Worst I have ever had was 5 minutes, but subsequent builds were reduced to under a minute due to build cache, creating multi-stage builds, and keeping the layers thin and optimizing the .dockerignore

People doing all the work of dependency fetches, code builds, and test execution inside ephemeral environments never designed for building software within.

Re: Faster Docker builds using a remote BuildKit instance

#36
There are a few tragedies in the Docker story, but at least two are specifically tied to naming things. First, Swarm (mode) because by the time they released Swarm (mode) the world had already taken a collective dump on Swarm (the proof-of-concept). Even in 2024 most of the time people talk about Swarm and start dumping on it they're actually talking about the proof-of-concept architecture. Second, they should never have called the subcommand "build." It isn't building anything. In this case "build" is performing a packaging step with very raw tools. But the minute they called it build people started literally building software INSIDE intermediate container layers on the way to assembling a packaged container. Dockerfile is about as weak of a build tool as you could possibly ask for. Zero useful features with respect to building software. But Docker named it "build" and now we've got Dockerfile calling compilation steps, test commands, and dependency retrieval steps.

Re: Faster Docker builds using a remote BuildKit instance

#38

There are a few tragedies in the Docker story, but at least two are specifically tied to naming things. First, Swarm (mode) because by the time they released Swarm (mode) the world had already taken a collective dump on Swarm (the proof-of-concept). Even in 2024 most of the time people talk about Swarm and start dumping on it they're actually talking about the proof-of-concept architecture. Second, they should never…

I can just imagine someone inside Docker, inc. saying “the name is not important” or over and over again until one day they ship it. They’ll never know what they missed out on.

Re: Faster Docker builds using a remote BuildKit instance

#39

There are a few tragedies in the Docker story, but at least two are specifically tied to naming things. First, Swarm (mode) because by the time they released Swarm (mode) the world had already taken a collective dump on Swarm (the proof-of-concept). Even in 2024 most of the time people talk about Swarm and start dumping on it they're actually talking about the proof-of-concept architecture. Second, they should never…

What is the alternative? Like one pipeline step running Gradle or make or something and then copying the result into your container that's basically an apt-get and nothing else?

Re: Faster Docker builds using a remote BuildKit instance

#40

Earlier quoted context omitted.

> we buy one or two very expensive machines that run CI and builds This unfortunately does not work for orgs that have, say, more than 20 engineers. The core issue is that once you have a test suite large enough to have ~30 shards, you only need one engineer `git push`ing once to saturate those 1-2 expensive machines you've got sitting in the office. The CI workload is quite amenable to "serverless" when you get to a…

I can get a 48 core/96 threads dedicated server for 200€ a month on hetzner. The cheapest EC2 with that comes close costs 2€ per hour. I can get nearly 10 hetzner servers that run consistently for that price. Obviously the dedicated machines are not rentable per hour, but the cloud is so much more expensive.

Very much this. I’ve overseen this process for one of my clients and we’ve seen build + deploy times go from 5-10 minutes down to around 1-2m. This was down to increased performance, improved caching, and being about to cut out some minor workflow setup steps.

So 10x cheaper and 5x the performance.

Still using GitHub Actions, but now just using self-hosted runners.

Post reply on HN