Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A faster path to container images in Bazel

tweag.io

Post reply on HN
Print view View original

31–40 of 60 posts

Re: A faster path to container images in Bazel

#31

Post by delichon »

cyberax wrote: post #29
slekker wrote: post #18

Earlier quoted context omitted.

As I've read through the post, seeing phrases like "Why this matters for performance", usage of em-dashes and lists/bullet points, screams AI written to me. I appreciate you saying it wasn't, but such is the fate of who wrote this to write like LLMs do nowadays. I also liked to use em-dashes and bullet lists but am consciously avoiding them now.

I interviewed a guy from Microsoft who was working on AI, and he literally speaks like this. Like, using the words "leverage", "matters for...", "as for", and so on. And you could almost hear him doing the bullet points. When you work with AI a lot, it changes your vocabulary.

That's absurd emdash I work with AI constantly and have noticed no such durable lexical shift.

Re: A faster path to container images in Bazel

#32

Post by boronine »

cyberax wrote: post #7

I'm struggling with the caching right now. I'm trying to switch from the Github actions to just running stuff in containers, and it works. Except for caching. Buildkit from Docker is just a pure bullshit design. Instead of the elegant layer-based system, there's now two daemons that fling around TAR files. And for no real reason that I can discern. But the worst thing is that the caching is just plain broken.

I went down this rabbit hole before, you have to ignore all the recommended approaches. The real solution is to have a build server with a global Docker install and a script to prune cache when the disk usage goes above a certain percentage. Cache is local and instant. Pushing and pulling cache images is an insane solution.

Re: A faster path to container images in Bazel

#33

Post by bmitch3020 »

> The current recommendation is rules_oci, which takes the opposite approach: use only off‑the‑shelf tools...

I'm the author of one of those off the shelf tools, and the rules_oci decision here always struck me as a bit unusual. OCI is a relatively easy spec with a number of libraries that implement it. Instead of creating a custom build command that leveraged those libraries to be an efficient build tool, they found commands that could be leveraged even if image building wasn't their design.

It looks like rules_img is taking that other path with their own build command based on the go-containerregistry library. I wish them all the best with their effort.

That said, if all you need to do is add a layer to an existing base, there are tools like crane [0] and regctl [1] that do that today.

The reason other build tools typically pull the base image first is to support "RUN" build steps that execute a command inside of a container and store the filesystem changes in a new layer. If that functionality is ever added to rules_img, I expect it to have the same performance as other build tools.

[0]: https://github.com/google/go-containerregistry/blob/main/cmd...

[1]: https://regclient.org/cli/regctl/image/mod/

Re: A faster path to container images in Bazel

#34

Post by paulddraper »

cyberax wrote: post #28
paulddraper wrote: post #21

Earlier quoted context omitted.

Huh? Each layer is a tarball. So build your tarballs (concurrently!), and then add some metadata to make an image. From your comment elsewhere it seems maybe you are expecting the docker build paradigm of running a container and snapshotting it at various stages. That is messy and has a number of limitations — not the least of which is cross-compilation. Reproducibility being another. But in any case, that definitely…

I don't quite understand how it handles running binaries then. For example, I want to do `bash -c "ls -la /"`. How would it run this command? It needs to assemble the filesystem at this point in the build process. I guess the answer for Bazel is "don't do this"? Docker handles cross-compilation by using emulators, btw.

> “don’t do this”

Yes. The Bazel way use to produce binaries, files, directories, and then create an image “directly” from these.

Much as you would create a JAR or ZIP or DEB.

This is (1) fast (2) small and (3) more importantly reproducible. Bazel users want their builds to produce artifacts that are exactly the same, for a number of reasons. Size is also nice…do you really need ls and dozens of other executables in your containerized service?

Most Docker users don’t care about reproducibility. They’ll apt-get install and get one version today and another version tomorrow.

Good? Bad? That’s a value judgement. But Bazel users have fundamentally different objectives.

> emulators

Yeah emulators is the Docker solution for producing images of different architectures.

Since Bazel doesn’t run commands as a running container, it skips that consideration entirely.

Re: A faster path to container images in Bazel

#35

Post by bmitch3020 »

cyberax wrote: post #7

I'm struggling with the caching right now. I'm trying to switch from the Github actions to just running stuff in containers, and it works. Except for caching. Buildkit from Docker is just a pure bullshit design. Instead of the elegant layer-based system, there's now two daemons that fling around TAR files. And for no real reason that I can discern. But the worst thing is that the caching is just plain broken.

Buildkit can be very efficient at caching, but you need to design your image build around it. Once any step encounters a cache miss, all remaining steps will too.

I'd also avoid loading the result back into the docker daemon unless you really need it there. Buildkit can output directly to a registry, or an OCI Layout, each of which will maintain the image digest and support multi-platform images (admittedly, those problems go away with the containerd storage changes happening, but it's still an additional export/import that can be skipped).

All that said, I think caching is often the wrong goal. Personally, I want reproducible builds, and those should bypass any cache to verify each step always has the same output. Also, when saving the cache, every build caches every step, even if they aren't used in future builds. As a result, for my own projects, the net result of adding a cache could be slower builds.

Instead of catching the image build steps, I think where we should be spending a lot more effort is in creating local proxies of upstream dependencies, removing the network overhead of pulling dependencies on every build. Compute intensive build steps would still be slow, but a significant number of image builds could be sped up with a proxy at the CI server level without tuning builds individually.

Re: A faster path to container images in Bazel

#36

Post by cyberax »

bmitch3020 wrote: post #35
cyberax wrote: post #7

I'm struggling with the caching right now. I'm trying to switch from the Github actions to just running stuff in containers, and it works. Except for caching. Buildkit from Docker is just a pure bullshit design. Instead of the elegant layer-based system, there's now two daemons that fling around TAR files. And for no real reason that I can discern. But the worst thing is that the caching is just plain broken.

Buildkit can be very efficient at caching, but you need to design your image build around it. Once any step encounters a cache miss, all remaining steps will too. I'd also avoid loading the result back into the docker daemon unless you really need it there. Buildkit can output directly to a registry, or an OCI Layout, each of which will maintain the image digest and support multi-platform images (admittedly, those pr…

> Buildkit can be very efficient at caching, but you need to design your image build around it.

Well, that's what I've been trying to do. And failing, because it simply doesn't work.

> I'd also avoid loading the result back into the docker daemon unless you really need it there.

I need Docker to provide me a reproducible environment to run lints, inspections, UI tests and so on. These images are quite massive. And because caching in Docker is broken, they were getting rebuilt every time we did a push.

Well. I switched to Podman and podman-compose. Now they do get cached, and the build time is within ~1 min with the help of the GHA cache.

And yes, my deployment builds are produced without any caching.

Re: A faster path to container images in Bazel

#37

Post by cyberax »

paulddraper wrote: post #34
cyberax wrote: post #28

Earlier quoted context omitted.

I don't quite understand how it handles running binaries then. For example, I want to do `bash -c "ls -la /"`. How would it run this command? It needs to assemble the filesystem at this point in the build process. I guess the answer for Bazel is "don't do this"? Docker handles cross-compilation by using emulators, btw.

> “don’t do this” Yes. The Bazel way use to produce binaries, files, directories, and then create an image “directly” from these. Much as you would create a JAR or ZIP or DEB. This is (1) fast (2) small and (3) more importantly reproducible. Bazel users want their builds to produce artifacts that are exactly the same, for a number of reasons. Size is also nice…do you really need ls and dozens of other executables in…

> Size is also nice…do you really need ls and dozens of other executables in your containerized service?

Yeah, I do. For debugging mostly :(

> Most Docker users don’t care about reproducibility. They’ll apt-get install and get one version today and another version tomorrow.

Ubuntu has daily snapshots. Not great, but works reasonably well. I tried going down the Nix route, but my team (well, and also myself) struggled with it.

I'd love to have fully bit-for-bit reproducible builds, but it's too complicated with the current tooling. Especially for something like mobile iOS apps (blergh).

Re: A faster path to container images in Bazel

#40

Post by rienbdj »

mgaunard wrote: post #19

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top. I'm not too surprised that out of the box docker images exhibit more of this. W…

What can used instead for a large multilanguage repo where we want to build every commit?

Genuine question - also find Bazel frustrating at times.

Post reply on HN