Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BuildKit: Docker's Hidden Gem That Can Build Almost Anything

tuananh.net

Post reply on HN
Print view View original

31–40 of 80 posts

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#31

Post by verdverm »

kodama-lens wrote: post #29
verdverm wrote: post #14

BuildKit also comes with a lot of pain. Dagger (a set of great interfaces to BuildKit in many languages) is working to remove it. Even their BuildKit maintainers think it's a good idea. BuildKit is very cool tech, but painful to run at volume Fun gotchya in BuildKit direct versus Dockerfiles, is the map iteration you loaded those ENV vars into consistent? No, that's why your cache keeps getting busted. You can't do t…

I switched our entire container build setup to buildkit. No kaniko, no buildah, no dind. The great part is that you can split buildkitd and the buildctl. Everything runs in its own docker runner. New buildkitd service for every job. Caching only via buildkit native cache export. Output format oci image compressed with zstd. Works pretty great so far, same or faster builds and we now create multi arch images. All on r…

That's pretty cool, rootless would be nice, but more effort than we see in ROI currently. I'm using the Dagger SDK directly, no CLI or modules.

Had to recently make it so multiple versions can run on the same host, such that as developers change branches, which may be on different IaC'd versions (we launch on demand), we don't break LTS release branches.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#32

Post by cyberax »

mid-kid wrote: post #26
cyberax wrote: post #25

Buildkit... It sounds great in theory, but it JustDoesn'tWork(tm). Its caching is plain broken, and the overhead of transmitting the entire build state to the remote computer every time is just busywork for most cases. I switched to Podman+buildah as a result, because it uses the previous dead simple Docker layered build system. If you don't believe me, try to make caching work on Github with multi-stage images. Just…

How do you use buildah? with dockerfiles? I find that buildah is sort of unbearably slow when using dockerfiles...

It has a braindead cache checking, I've fixed it locally and I'm cleaning it up for the upstream submission. But otherwise, it's always faster for me than Buildkit.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#33

Post by skrtskrt »

kylegalbraith wrote: post #28

After building Depot [0] for the past three years, I can say I have a ton of scar tissue from running BuildKit to power our remote container builders for thousands of organizations. It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason abo…

> It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason about, or in the worst case, terrifying to touch.

This is true of packaging and build systems in general. They are often the passion projects of one or a handful of people in an organization - by the time they have active outside development, those idiosyncratic concepts are already ossified.

It's really rare to see these sorts of projects decomposed into building blocks even just having code organization that helps a newcomer understand. Despite all the code being out in public, all the important reasoning about why certain things are the way they are is trapped inside a few dev's heads.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#35

Post by cyberax »

hanikesn wrote: post #30
cyberax wrote: post #25

Buildkit... It sounds great in theory, but it JustDoesn'tWork(tm). Its caching is plain broken, and the overhead of transmitting the entire build state to the remote computer every time is just busywork for most cases. I switched to Podman+buildah as a result, because it uses the previous dead simple Docker layered build system. If you don't believe me, try to make caching work on Github with multi-stage images. Just…

Why would you use the horrible GHA cache and not a much more efficient registry based cache?

Registry cache...

It's yet one more incomprehensible Buildkit decision. The original Docker builder had a very simple cache system: it computed the layer hash and then checked the registry for its presence. Simple content-addressable caching.

Buildkit can NOT do this. Instead, it uses a single image as a dumping ground for the caches. If you have two builders using the same image, they'll step on each other's toes. GHA at least side-steps this.

But I tried the registry cache, and it didn't improve anything. So far, I was not able to get caching to work with multi-stage builds at all. There are open issues for that, dating back to 2020.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#36

Post by Izkata »

craftkiller wrote: post #5
moochmooch wrote: post #3

unfortunately, make is more well written software. I think ultimately Dockerfile was a failed iteration of Makefile. YAML & Dockerfile are poor interfaces for these types of applications. The code first options are quite good these days, but you can get so far with make & other legacy tooling. Docker feels like a company looking to sell enterprise software first and foremost, not move the industry standard forward gr…

Along similar lines, when I was reading the article I was thinking "this just sounds like a slightly worse version of nix". Nix has the whole content addressed build DAG with caching, the intermediate language, and the ability to produce arbitrary outputs, but it is functional (100% of the inputs must be accounted for in the hashes/lockfile, as opposed to Docker where you can run commands like `apk add firefox` which…

> whereas nix will always produce the same output for a given input.

If they didn't take shortcuts. I don't know if it's been fixed, but at one point Vuze in nix pulled in an arbitrary jar file from a URL. I had to dig through it because the jar had been updated at some point but not the nix config and it was failing at an odd place.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#37

Post by moochmooch »

kccqzy wrote: post #6
moochmooch wrote: post #3

unfortunately, make is more well written software. I think ultimately Dockerfile was a failed iteration of Makefile. YAML & Dockerfile are poor interfaces for these types of applications. The code first options are quite good these days, but you can get so far with make & other legacy tooling. Docker feels like a company looking to sell enterprise software first and foremost, not move the industry standard forward gr…

Make is timestamp based. That is a thoroughly out-of-date approach only suitable for a single computer. You want distributed hash-based caching in the modern world.

so use Bazel or buck2 if you need an iteration on make's handling of changed files. Bazel is much more serious of a project than buildkit. I'm not saying make is more functional that buildkit (it might be to some), I'm saying its better written software than buildkit. two separate things

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#38

Post by matheus-rr »

The --mount=type=cache for package managers is genuinely transformative once you figure it out. Before that, every pip install or apt-get in a Dockerfile was either slow (no caching) or fragile (COPY requirements.txt early and pray the layer cache holds).

What nobody tells you is that the cache mount is local to the builder daemon. If you're running builds on ephemeral CI instances, those caches are gone every build and you're back to square one. The registry cache backend exists to solve this but it adds enough complexity that most teams give up and just eat the slow builds.

The other underrated BuildKit feature is the ssh mount. Being able to forward your SSH agent into a build step without baking keys into layers is the kind of thing that should have been in Docker from day one. The number of production images I've seen with SSH keys accidentally left in intermediate layers is genuinely concerning.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#39

Post by scuff3d »

croes wrote: post #27
scuff3d wrote: post #19

Earlier quoted context omitted.

The "This is the key insight -" or "x is where it gets practical -", are dead give aways too. If I wanted an LLMs explanation of how it works, I can ask an LLM. When I see articles like this I'm expecting an actual human expert

And waste time and energy again to get a similar result?

An article written by an expert is nothing like this. You might be able to get something similar out of an LLM but it's gonna take a lot more effort then was out into this.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#40

Post by literalAardvark »

moochmooch wrote: post #37
kccqzy wrote: post #6

Earlier quoted context omitted.

Make is timestamp based. That is a thoroughly out-of-date approach only suitable for a single computer. You want distributed hash-based caching in the modern world.

so use Bazel or buck2 if you need an iteration on make's handling of changed files. Bazel is much more serious of a project than buildkit. I'm not saying make is more functional that buildkit (it might be to some), I'm saying its better written software than buildkit. two separate things

Bazel just seems so... Academic. I can't make heads or tails of it.

Compared to a Dockerfile it's just too hard to follow

Post reply on HN