Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A faster path to container images in Bazel

tweag.io

Post reply on HN
Print view View original

21–30 of 60 posts

Re: A faster path to container images in Bazel

#21

Post by paulddraper »

cyberax wrote: post #7

I'm struggling with the caching right now. I'm trying to switch from the Github actions to just running stuff in containers, and it works. Except for caching. Buildkit from Docker is just a pure bullshit design. Instead of the elegant layer-based system, there's now two daemons that fling around TAR files. And for no real reason that I can discern. But the worst thing is that the caching is just plain broken.

Huh?

Each layer is a tarball.

So build your tarballs (concurrently!), and then add some metadata to make an image.

From your comment elsewhere it seems maybe you are expecting the docker build paradigm of running a container and snapshotting it at various stages.

That is messy and has a number of limitations — not the least of which is cross-compilation. Reproducibility being another. But in any case, that definitely not what these rules are trying to do.

Re: A faster path to container images in Bazel

#22

Post by paulddraper »

This is smart.

Container layers are so large that moving them around is heavy.

So defer that part for the non-hermetic push/load parts of the process, while retaining heremticity/reproducibility.

You can sort of think of it like the IO monad in Haskell…defer it all until the impure end.

Re: A faster path to container images in Bazel

#23

Post by paulddraper »

mgaunard wrote: post #19

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top. I'm not too surprised that out of the box docker images exhibit more of this. W…

To be clear, when you say “they’re fixing this”…the Bazel maintainers have nothing to do with this.

Bazel is a general purpose tool like Make. But with caching and sandboxing and different syntax.

Make is no less focused on Docker than Bazel is.

Unlike Make however, Bazel does make it easy to share rule sets.

But you don’t need to use other people’s Bazel rule sets any more than you need to use other people’s Make recipes.

This author has a clever way to minimize needing to touch layers at all.

Re: A faster path to container images in Bazel

#24

Post by cpuguy83 »

cyberax wrote: post #11
klysm wrote: post #9

Earlier quoted context omitted.

The layers are tar files, I’m confused what behavior you actually want that isn’t supported.

The original Docker (and the current Podman) created each layer as an overlay filesystem. So each layer was essentially an ephemeral container. If a build failed, you could actually just run the last successful layer with a shell and see what's wrong. More importantly, the layers were represented as directories on the host system. So when you wanted to run something in the final container, Docker just needed to reass…

Buildkit didn't break anything here except that it each individual build step is no longer exposed as a runnable image in docker. That was unfortunate, but you can actually have buildkit run a command in that filesystem these days, and buildx now even exposes a DAP interface.

Buldkit is far more efficient than the old model.

Re: A faster path to container images in Bazel

#25

Post by liuliu »

paulddraper wrote: post #23
mgaunard wrote: post #19

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top. I'm not too surprised that out of the box docker images exhibit more of this. W…

To be clear, when you say “they’re fixing this”…the Bazel maintainers have nothing to do with this. Bazel is a general purpose tool like Make. But with caching and sandboxing and different syntax. Make is no less focused on Docker than Bazel is. Unlike Make however, Bazel does make it easy to share rule sets. But you don’t need to use other people’s Bazel rule sets any more than you need to use other people’s Make re…

rules_oci (and bunch of rules_* under bazelbuild / bazel-contrib org on GitHub) is Bazel recommeded rule sets.

I don't agree with your parent comment about Bazel, but your comment is not fair too. Bazel tries to be better build tool so it took on responsibility on registry / rules_* and get critics for it is a fair game.

The "bloated Bazel" blame is not fair too, but I think somewhat understandable. If you ever going to only do JavaScript, bun or other package manager is enough and "lighter-weight". Same goes to uv + Python bundle. Bazel only shines if you are dealing with your C++ mess and even there, people prefer CMake for reasons beyond me.

Re: A faster path to container images in Bazel

#26

Post by gugagore »

mgaunard wrote: post #19

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top. I'm not too surprised that out of the box docker images exhibit more of this. W…

For some more depth into the "bloat" of bazel, I like this reference: https://fzakaria.com/2024/02/27/hermetic-but-at-what-cost

Re: A faster path to container images in Bazel

#27

Post by cyberax »

cpuguy83 wrote: post #24
cyberax wrote: post #11

Earlier quoted context omitted.

The original Docker (and the current Podman) created each layer as an overlay filesystem. So each layer was essentially an ephemeral container. If a build failed, you could actually just run the last successful layer with a shell and see what's wrong. More importantly, the layers were represented as directories on the host system. So when you wanted to run something in the final container, Docker just needed to reass…

Buildkit didn't break anything here except that it each individual build step is no longer exposed as a runnable image in docker. That was unfortunate, but you can actually have buildkit run a command in that filesystem these days, and buildx now even exposes a DAP interface. Buldkit is far more efficient than the old model.

Buildkit is still a separate system, unlike the old builder. So you get that extra step of importing the result back.

And since it's a separate system, there are also these strange limitations. For example, I can't just cache pre-built images in an NFS directory and then just push them into the Buildkit context. There's simply no command for it. Buildkit can only pull them from a registry.

> Buldkit is far more efficient than the old model.

I've yet to see it work faster than podman+buildah. And it's also just plain buggy. Caching for multi-stage and/or parallel builds has been broken since the beginning. The Docker team just ignores it and closes the bugs: https://github.com/moby/buildkit/issues/1981 https://github.com/moby/buildkit/issues/2274 https://github.com/moby/buildkit/issues/2279

I understand why. I tried to debug it, and simply getting it running under a debugger is an adventure.

So far, I found that switching to podman+podman-compose is a better solution. At least my brain is good enough to understand them completely, and contribute fixes if needed.

Re: A faster path to container images in Bazel

#28

Post by cyberax »

paulddraper wrote: post #21
cyberax wrote: post #7

I'm struggling with the caching right now. I'm trying to switch from the Github actions to just running stuff in containers, and it works. Except for caching. Buildkit from Docker is just a pure bullshit design. Instead of the elegant layer-based system, there's now two daemons that fling around TAR files. And for no real reason that I can discern. But the worst thing is that the caching is just plain broken.

Huh? Each layer is a tarball. So build your tarballs (concurrently!), and then add some metadata to make an image. From your comment elsewhere it seems maybe you are expecting the docker build paradigm of running a container and snapshotting it at various stages. That is messy and has a number of limitations — not the least of which is cross-compilation. Reproducibility being another. But in any case, that definitely…

I don't quite understand how it handles running binaries then. For example, I want to do `bash -c "ls -la /"`. How would it run this command? It needs to assemble the filesystem at this point in the build process.

I guess the answer for Bazel is "don't do this"? Docker handles cross-compilation by using emulators, btw.

Re: A faster path to container images in Bazel

#29

Post by cyberax »

slekker wrote: post #18
Xophmeister wrote: post #14

Earlier quoted context omitted.

It’s not. It’s been through several editing rounds. (I was one of the editors.) In theory, we don’t have a problem with AI generated content if it meets our high editorial requirements, but all Tweag technical blogs go through a rigorous, manual review and editing process to keep standards high.

As I've read through the post, seeing phrases like "Why this matters for performance", usage of em-dashes and lists/bullet points, screams AI written to me. I appreciate you saying it wasn't, but such is the fate of who wrote this to write like LLMs do nowadays. I also liked to use em-dashes and bullet lists but am consciously avoiding them now.

I interviewed a guy from Microsoft who was working on AI, and he literally speaks like this.

Like, using the words "leverage", "matters for...", "as for", and so on. And you could almost hear him doing the bullet points.

When you work with AI a lot, it changes your vocabulary.

Re: A faster path to container images in Bazel

#30

Post by [deleted] »

cyberax wrote: post #29
slekker wrote: post #18

Earlier quoted context omitted.

As I've read through the post, seeing phrases like "Why this matters for performance", usage of em-dashes and lists/bullet points, screams AI written to me. I appreciate you saying it wasn't, but such is the fate of who wrote this to write like LLMs do nowadays. I also liked to use em-dashes and bullet lists but am consciously avoiding them now.

I interviewed a guy from Microsoft who was working on AI, and he literally speaks like this. Like, using the words "leverage", "matters for...", "as for", and so on. And you could almost hear him doing the bullet points. When you work with AI a lot, it changes your vocabulary.

[deleted]
Post reply on HN