Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

BuildKit: Docker's Hidden Gem That Can Build Almost Anything

tuananh.net

Post reply on HN
Print view View original

41–50 of 80 posts

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#41

Post by mikepurvis »

kylegalbraith wrote: post #28

After building Depot [0] for the past three years, I can say I have a ton of scar tissue from running BuildKit to power our remote container builders for thousands of organizations. It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason abo…

I introduced Depot at my org a few months ago and I've been very happy with it. Conceptually it's simple: a container builder that starts warm with all your previously built layers right there, same as it would be running local builds. But a lot goes into making it actually run smoothly, and the performance-focused breakdown that shows where steps depend on each other and how much time each is taking is great.

It's clear a ton of care has gone into the product, and I also appreciated you personally jumping onto some of my support tickets when I was just getting things off the ground.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#42

Post by mikepurvis »

skrtskrt wrote: post #33
kylegalbraith wrote: post #28

After building Depot [0] for the past three years, I can say I have a ton of scar tissue from running BuildKit to power our remote container builders for thousands of organizations. It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason abo…

> It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason about, or in the worst case, terrifying to touch. This is true of packaging and build systems in general. They are often the passion projects of one or a handful of people in an organization - by the time they have active outside development, those idiosyncratic concepts are…

As someone who has worked in the space for a while and been heavily exposed to nix, bazel, cmake, bake, and other systems, and also been in that "passion project" role, I think what I've found is that these kinds of systems are just plain hard to talk about. Even the common elements like DAGs cause most people's eyes to immediately glaze over.

Managers and executives are happy to hear that you made the builds faster or more reliable, so the infra people who care about this kind of thing don't waste time on design docs and instead focus on getting to a minimum prototype that demonstrates those improved metrics. Once you have that, then there's buy-in and the project is made official... but by then the bones have already been set in place, so design documentation ends up focused on the more visible stuff like user interface, storage formats, etc.

OTOH, bazel (as blaze) was a very intentionally designed second system at Google, and buildx/buildkit is similarly a rewrite of the container builder for Docker, so both of them should have been pretty free of accidental engineering in their early phases.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#43

Post by candiddevmike »

matheus-rr wrote: post #38

The --mount=type=cache for package managers is genuinely transformative once you figure it out. Before that, every pip install or apt-get in a Dockerfile was either slow (no caching) or fragile (COPY requirements.txt early and pray the layer cache holds). What nobody tells you is that the cache mount is local to the builder daemon. If you're running builds on ephemeral CI instances, those caches are gone every build…

I hate the nanny state behavior of docker build and not being allowed to modify files/data outside of the build container and cache, like having a NFS mount for sharing data in the build or copying files out of the build.

Let me have side effects, I'm a consenting adult and understand the consequences!!!

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#44

Post by kccqzy »

moochmooch wrote: post #37
kccqzy wrote: post #6

Earlier quoted context omitted.

Make is timestamp based. That is a thoroughly out-of-date approach only suitable for a single computer. You want distributed hash-based caching in the modern world.

so use Bazel or buck2 if you need an iteration on make's handling of changed files. Bazel is much more serious of a project than buildkit. I'm not saying make is more functional that buildkit (it might be to some), I'm saying its better written software than buildkit. two separate things

Oh I love Bazel. The problem is that it’s harder to adopt for teams used to just using make. For a particular project at work, I argued unsuccessfully for switching from plain make to bazel, and it ended up switching to cmake.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#45

Post by nl »

skrtskrt wrote: post #33
kylegalbraith wrote: post #28

After building Depot [0] for the past three years, I can say I have a ton of scar tissue from running BuildKit to power our remote container builders for thousands of organizations. It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason abo…

> It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason about, or in the worst case, terrifying to touch. This is true of packaging and build systems in general. They are often the passion projects of one or a handful of people in an organization - by the time they have active outside development, those idiosyncratic concepts are…

> This is true of packaging and build systems in general. They are often the passion projects of one or a handful of people in an organization

This is a very insightful comment

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#46

Post by tuananh »

kylegalbraith wrote: post #28

After building Depot [0] for the past three years, I can say I have a ton of scar tissue from running BuildKit to power our remote container builders for thousands of organizations. It looks and sounds incredibly powerful on paper. But the reality is drastically different. It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason abo…

Thanks for the insight Kyle. If Depot can open-source it, that would be amazing for the community.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#47

Post by throwway120385 »

mikepurvis wrote: post #42
skrtskrt wrote: post #33

Earlier quoted context omitted.

> It's a big glob of homegrown thoughts and ideas. Some of them are really slick, like build deduplication. Others are clever and hard to reason about, or in the worst case, terrifying to touch. This is true of packaging and build systems in general. They are often the passion projects of one or a handful of people in an organization - by the time they have active outside development, those idiosyncratic concepts are…

As someone who has worked in the space for a while and been heavily exposed to nix, bazel, cmake, bake, and other systems, and also been in that "passion project" role, I think what I've found is that these kinds of systems are just plain hard to talk about. Even the common elements like DAGs cause most people's eyes to immediately glaze over. Managers and executives are happy to hear that you made the builds faster…

I don't think you can ever get away from accidental engineering in build systems because as soon as they find their niche something new comes along to disrupt it. Even with something homegrown out of shell scripts and directory trees the boss will eventually ask you to do something that doesn't fit well with your existing concepts.

A build system is meant to yield artifacts, run tools, parallelize things, calculate dependencies, download packages, and more. And these are all things that have some algorithmic similarity which is a kind of superficial similarity in that the failure modes and the exact systems involved are often dramatically different. I don't know that you can build something that is that all-encompassing without compromising somewhere.

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#48

Post by verdverm »

xyzzy_plugh wrote: post #15
ricardobeat wrote: post #12

Earlier quoted context omitted.

> so two docker builds can end up with the same hash but different output The cache key includes the state of the filesystem so I don’t think that would ever be true. Regardless, the purpose of the tool is to generate [layer] images to be reused, exactly to avoid the pitfalls of reproducible builds, isn’t it? In the context of the article, what makes builds reproducible is the shared cache.

It's not reproducible then, it's simply cached. It's a valid approach but there's tradeoffs of course.

it's not an either or, it can be reproducible and cached

similarly, nix cannot guarantee reproducibility if the user does things to break that possibility

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#49

Post by verdverm »

craftkiller wrote: post #20
ricardobeat wrote: post #12

Earlier quoted context omitted.

> so two docker builds can end up with the same hash but different output The cache key includes the state of the filesystem so I don’t think that would ever be true. Regardless, the purpose of the tool is to generate [layer] images to be reused, exactly to avoid the pitfalls of reproducible builds, isn’t it? In the context of the article, what makes builds reproducible is the shared cache.

Ah you're right, the hash wouldn't be the same but a Dockerfile could produce different outputs on different machines whereas nix will produce identical output on different machines.

If you are building a binary on different arches, it will not be the same. I have many container builds that I can run while disabling the cache and get the same hash/bytes in the end, i.e. reproducible across machines, which also requires whatever you build inside be byte reproducible (like Go)

Re: BuildKit: Docker's Hidden Gem That Can Build Almost Anything

#50

Post by verdverm »

jasonpeacock wrote: post #7
craftkiller wrote: post #5

Earlier quoted context omitted.

Along similar lines, when I was reading the article I was thinking "this just sounds like a slightly worse version of nix". Nix has the whole content addressed build DAG with caching, the intermediate language, and the ability to produce arbitrary outputs, but it is functional (100% of the inputs must be accounted for in the hashes/lockfile, as opposed to Docker where you can run commands like `apk add firefox` which…

You can network-jail your builds to prevent pulling from external repos and force the build environment to define/capture its inputs.

just watch out for built at timestamps
Post reply on HN