Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BuildKit: Docker's Hidden Gem That Can Build Almost Anything

tuananh.net

Post reply on HN
Print view View original

61–70 of 80 posts

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#61

Post by kylegalbraith »

mikepurvis wrote: post #41
kylegalbraith wrote: post #28

After building Depot [0] for the past three years, I can say I have a ton of scar tissue from running BuildKit to power our remote container builders for thousands of organizations. It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason abo…

I introduced Depot at my org a few months ago and I've been very happy with it. Conceptually it's simple: a container builder that starts warm with all your previously built layers right there, same as it would be running local builds. But a lot goes into making it actually run smoothly, and the performance-focused breakdown that shows where steps depend on each other and how much time each is taking is great. It's c…

Thank you for the very kind words and for your support. Depot is full of incredible people who love helping others. So while you might see me on a ticket from time to time, it’s really an entire team that is behind everything we do.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#62

Post by kylegalbraith »

tuananh wrote: post #46
kylegalbraith wrote: post #28

After building Depot [0] for the past three years, I can say I have a ton of scar tissue from running BuildKit to power our remote container builders for thousands of organizations. It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason abo…

Thanks for the insight Kyle. If Depot can open-source it, that would be amazing for the community.

Thank you for the post! It’s well done and you captured a lot of the concepts in BuildKit in an easy to understand way. Not an easy thing to do at all.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#63

Post by kylegalbraith »

bmitch3020 wrote: post #9

I don't use buildkit for artifacts, but I do like to output images to an OCI Layout so that I can finish some local checks and updates before pushing the image to a registry. But the real hidden power of buildkit is the ability to swap out the Dockerfile parser. If you want to see that in action, look at this Dockerfile (yes, that's yaml) used for one of their hardened images: https://github.com/docker-hardened-image…

I agree on both fronts! BuildKit frontends are not very well known but can be very powerful if you know how they work and how BuildKit transforms them.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#64

Post by dilyevsky »

kccqzy wrote: post #44
moochmooch wrote: post #37

Earlier quoted context omitted.

so use Bazel or buck2 if you need an iteration on make's handling of changed files. Bazel is much more serious of a project than buildkit. I'm not saying make is more functional that buildkit (it might be to some), I'm saying its better written software than buildkit. two separate things

Oh I love Bazel. The problem is that it’s harder to adopt for teams used to just using make. For a particular project at work, I argued unsuccessfully for switching from plain make to bazel, and it ended up switching to cmake.

Now with AI bazel maintenance is almost entirely painless experience. I have fewer issues with it than the standard Go toolchain and C++ experience was always quite smooth.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#65

Post by XYen0n »

Izkata wrote: post #36
craftkiller wrote: post #5

Earlier quoted context omitted.

Along similar lines, when I was reading the article I was thinking "this just sounds like a slightly worse version of nix". Nix has the whole content addressed build DAG with caching, the intermediate language, and the ability to produce arbitrary outputs, but it is functional (100% of the inputs must be accounted for in the hashes/lockfile, as opposed to Docker where you can run commands like `apk add firefox` which…

> whereas nix will always produce the same output for a given input. If they didn't take shortcuts. I don't know if it's been fixed, but at one point Vuze in nix pulled in an arbitrary jar file from a URL. I had to dig through it because the jar had been updated at some point but not the nix config and it was failing at an odd place.

This should result in a hash mismatch error rather than an output different from the previous one. If there is a way to locate the original jar file (hash matching), it will still produce the same output as before.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#66

Post by mkesper »

verdverm wrote: post #52
xyzzy_plugh wrote: post #51

Earlier quoted context omitted.

The difference is that you can blow the Nix cache away and reproduce it entirely. The same cannot be said for Docker.

That's not true Docker has a `--no-cache` flag, even easier than blowing it away, which you can also do with several built in commands or a rm -rf /var/lib/docker Perhaps worth revisiting: https://docs.docker.com/build/cache/

That will rebuild the cache from upstream but not reproducibly.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#67

Post by Orygin »

kodama-lens wrote: post #29
verdverm wrote: post #14

BuildKit also comes with a lot of pain. Dagger (a set of great interfaces to BuildKit in many languages) is working to remove it. Even their BuildKit maintainers think it's a good idea. BuildKit is very cool tech, but painful to run at volume Fun gotchya in BuildKit direct versus Dockerfiles, is the map iteration you loaded those ENV vars into consistent? No, that's why your cache keeps getting busted. You can't do t…

I switched our entire container build setup to buildkit. No kaniko, no buildah, no dind. The great part is that you can split buildkitd and the buildctl. Everything runs in its own docker runner. New buildkitd service for every job. Caching only via buildkit native cache export. Output format oci image compressed with zstd. Works pretty great so far, same or faster builds and we now create multi arch images. All on r…

That's my experience too. Coming from single threaded, single concurrent build Kaniko to parallel builds using Buildkit reduced build times by 2 to 3.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#68

Post by dewey »

kylegalbraith wrote: post #28

After building Depot [0] for the past three years, I can say I have a ton of scar tissue from running BuildKit to power our remote container builders for thousands of organizations. It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason abo…

Looks very interesting, am I reading it correctly that this is only available on AWS for "bring your own compute"?

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#69

Post by westurner »

thayne wrote: post #57
mikepurvis wrote: post #42

Earlier quoted context omitted.

As someone who has worked in the space for a while and been heavily exposed to nix, bazel, cmake, bake, and other systems, and also been in that "passion project" role, I think what I've found is that these kinds of systems are just plain hard to talk about. Even the common elements like DAGs cause most people's eyes to immediately glaze over. Managers and executives are happy to hear that you made the builds faster…

Blaze and bazel may have been intentionally designed, but it was designed for Google's needs, and it shows (at least from my observations of bazel, I don't have any experience with blaze). It is better now than it was, but it obviously was designed for a system where most dependencies are vendored, and worked better for languages that google used like c++, java, and python.

Blaze instead of make, ant, maven. But now there's cmake and ninjabuild. gn wraps ninjabuild wraps cmake these days fwiu.

Blaze is/was integrated with Omega scheduler, which is not open.

Bazel is open source.

By the time Bazel was open sourced, Twitter had pantsbuild and Facebook had buck.

OpenWRT's Makefiles are sufficient to build OpenWRT and the kernel for it. (GNU Make is still sufficient to build the Linux kernel today, in 2026.)

Make compares files to determine whether to rebuild them if they already exist; by comparing file modification time (mtime) unless the task name is in the .PHONY: list at the top of the Makefile. But the task names may not contain slashes or spaces.

`docker build` and so also BuildKit archive the build chroot after each build step that modifies the filesystem (RUN, ADD, COPY) as a cacheable layer identified by a hash of its content.

Other Dockerfile instructions add metadata: CMD, ENTRYPOINT, LABEL, ENV, ARG, WORKDIR, USER, EXPOSE , VOLUME .

The FROM instruction creates a build stage from scratch or from a different container layer.

Dockerfile added support for Multi-stage builds with multiple `FROM` instructions in 2017 (versions 17.05, 17.06CE).

`docker build` is now moby and there is also buildkit? `podman buildx` seems to work.

nerdctl supports a number of features that have not been merged back to docker or to podman.

> it obviously was designed for a system where most dependencies are vendored, and worked better for languages that google used like c++, java, and python.

Those were the primary languages at google at the time. And then also to build software? Make, shell scripts, python, that Makefile calls git which calls perl so perl has to be installed, etc.

Also gtests and gflags.

"Compiler Options Hardening Guide for C and C++" https://news.ycombinator.com/item?id=43551959 :

>> There are default gcc and/or clang compiler flags in distros' default build tools; e.g. `make` specifies additional default compiler flags (that e.g. cmake, ninja, gn, or bazel/buck/pants may not also specify for you).

Which CPU microarchitectures and flags are supported?

  ld.so --help | grep "supported"
  cat /proc/cpuinfo | grep -E '^(flags|bugs)'`
AVX-512 is in x86-64-v3. By utilizing features like AVX-512, we would save money (by utilizing features in processors newer than Pentium 4 (x86-64-v1)).

How to add an `-march=x86-64-v3` argument to every build?

How to add build flags to everything for something like x86-64-v4?

Which distros support consistent build parametrization to make adding global compiler build flags for multiple compilers?

- Gentoo USE flags

- rebuild a distro and commit to building the core and updates and testing and rawhide with your own compiler flags and package signatures and host mirrored package repos

- Intel Clear Linux was cancelled.

- CachyOS (x86-64-v3, x86-64-v4, Zen4)

- conda-forge?

Gentoo:

- ChromiumOS was built on gentoo and ebuild IIRC

- emerge app-portage/cpuid2cpuflags, CPU_FLAGS_X86=, specify -march=native for C/[C++] and also target-cpu=native for Rust in /etc/portage/make.conf

- "Gentoo x86-64-v3 binary packages available" (2024) https://news.ycombinator.com/item?id=39250609

Google, Facebook, and Twitter have a monorepo to build packages from.

Google had a monorepo at the time that blaze was written.

Twitter ("X") is moving from pantsbuild to blaze BUILD files.

TIL there is a buck2. How does facebook/buck2 compare to google/bazel (compare to what is known about blaze)?

Should I build containers (chroot fs archives) with ansible? Then there is no buildkit.

FWIW `podman-kube-play` can run some kubernetes yaml.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#70

Post by mikepurvis »

westurner wrote: post #69
thayne wrote: post #57

Earlier quoted context omitted.

Blaze and bazel may have been intentionally designed, but it was designed for Google's needs, and it shows (at least from my observations of bazel, I don't have any experience with blaze). It is better now than it was, but it obviously was designed for a system where most dependencies are vendored, and worked better for languages that google used like c++, java, and python.

Blaze instead of make, ant, maven. But now there's cmake and ninjabuild. gn wraps ninjabuild wraps cmake these days fwiu. Blaze is/was integrated with Omega scheduler, which is not open. Bazel is open source. By the time Bazel was open sourced, Twitter had pantsbuild and Facebook had buck. OpenWRT's Makefiles are sufficient to build OpenWRT and the kernel for it. (GNU Make is still sufficient to build the Linux kerne…

The ansible-in-containers thing is very much an unsolved problem. Basically right now you have three choices:

- install ansible in-band and run it against localhost (sucks because your playbook is in a final image layer; you might not want Python at all in the container)

- use packer with ansible as your provisioner and a docker container export, see: https://alex.dzyoba.com/blog/packer-for-docker/

- copy a previous stage's root into a subdirectory and then run ansible on that as a chroot, afterward copy the result back to a scratch container's root.

All of these options fall down when you're doing anything long-running though, because they can't work incrementally. As soon as you call ansible (or any other tool), then from Docker's point of view it's now a single step. This is really unfortunate because a Dockerfile is basically just shell invocations, and ansible gives a more structured and declarative-ish way to do shell type things.

I have wondered if a system like Dagger might be able to do a better job with this, basically break up the playbook programmatically into single task sub-playbooks and call each one in its own Dagger task/layer. This would allow ansible to retain most of its benefits while not being as hamstrung by the semantics of the caller. And it would be particularly nice for the case where the container is ultimately being exported to a machine image because then if you've defined everything in ansible you have a built-in story for freshening that deployed system later as the playbook evolves.

Post reply on HN