Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A faster path to container images in Bazel

tweag.io

Post reply on HN
Print view View original

51–60 of 60 posts

Re: A faster path to container images in Bazel

#51

Post by jakewins »

rienbdj wrote: post #40
mgaunard wrote: post #19

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top. I'm not too surprised that out of the box docker images exhibit more of this. W…

What can used instead for a large multilanguage repo where we want to build every commit? Genuine question - also find Bazel frustrating at times.

I tried Basel, Buck2 and Pants for a greenfield mono repo recently, Rust and Python heavy.

Of the three, I went with Buck2. Maybe just circumstance with Rust support being good and not built to replace Cargo?

Bazel was a huge pain - broke all standard tooling by taking over Cargos job, then unable to actually build most packages without massive multi-day patching efforts.

Pants seemed premature - front page examples from the docs didn’t work, apparently due to breaking changes in minor versions, and the Rust support is very very early days.

Buck2 worked out of the box exactly as claimed, leaves Cargo intact so all the tooling works.. I’m hopeful.

Previously I’ve used Make for polyglot monorepos.. but it requires an enormous amount of discipline from the team, so I’m very keen for a replacement with less foot guns

Re: A faster path to container images in Bazel

#52

Post by paulddraper »

__turbobrew__ wrote: post #42
paulddraper wrote: post #22

This is smart. Container layers are so large that moving them around is heavy. So defer that part for the non-hermetic push/load parts of the process, while retaining heremticity/reproducibility. You can sort of think of it like the IO monad in Haskell…defer it all until the impure end.

Is load not hermetic? Ideally you should be mirroring all layers you use as inputs to your OCI builds and pin SHA256 versions. Your caching will also have issues if you don’t pin versions. Push should also be idempotent, but not hermetic.

Load is like a push, but with a local daemon.

Re: A faster path to container images in Bazel

#53

Post by oftenwrong »

boronine wrote: post #32
cyberax wrote: post #7

I'm struggling with the caching right now. I'm trying to switch from the Github actions to just running stuff in containers, and it works. Except for caching. Buildkit from Docker is just a pure bullshit design. Instead of the elegant layer-based system, there's now two daemons that fling around TAR files. And for no real reason that I can discern. But the worst thing is that the caching is just plain broken.

I went down this rabbit hole before, you have to ignore all the recommended approaches. The real solution is to have a build server with a global Docker install and a script to prune cache when the disk usage goes above a certain percentage. Cache is local and instant. Pushing and pulling cache images is an insane solution.

What you are describing is basically remote buildkitd. That allows all of your docker builds to share a big cache. The cache-to/cache-from approach is of limited usefulness.

Re: A faster path to container images in Bazel

#54

Post by cyberax »

cpuguy83 wrote: post #44
cyberax wrote: post #27

Earlier quoted context omitted.

Buildkit is still a separate system, unlike the old builder. So you get that extra step of importing the result back. And since it's a separate system, there are also these strange limitations. For example, I can't just cache pre-built images in an NFS directory and then just push them into the Buildkit context. There's simply no command for it. Buildkit can only pull them from a registry. > Buldkit is far more effic…

Buildkit is integrated into dockerd the same way the old builder was. If you want a newer Buildkit you'll need to run it separately of course. I'm not quite sure I understand what you are trying to do with nfs there. But you can definitely export the cache to a local filesystem and import it with cache-from. You can also provide named contexts. "Buildkit can only pull them from a registry" is just plain false.

> Buildkit is integrated into dockerd the same way the old builder was. If you want a newer Buildkit you'll need to run it separately of course.

I don't think that the older builder created special containers for itself?

> I'm not quite sure I understand what you are trying to do with nfs there. But you can definitely export the cache to a local filesystem and import it with cache-from.

Which is dog-slow, because it squirts the cache through a socket. I have an NFS disk that can be used to cache the data directly. This was just one of the attempts to make it go faster.

> You can also provide named contexts.

Which can only refer to images that are built inside this particular buildkit or are pullable from a repo.

This is really all I want, a way to quickly reused the previous state saved in some format in Github Cache, NFS, or other storage.

Re: A faster path to container images in Bazel

#55

Post by cpuguy83 »

cyberax wrote: post #54
cpuguy83 wrote: post #44

Earlier quoted context omitted.

Buildkit is integrated into dockerd the same way the old builder was. If you want a newer Buildkit you'll need to run it separately of course. I'm not quite sure I understand what you are trying to do with nfs there. But you can definitely export the cache to a local filesystem and import it with cache-from. You can also provide named contexts. "Buildkit can only pull them from a registry" is just plain false.

> Buildkit is integrated into dockerd the same way the old builder was. If you want a newer Buildkit you'll need to run it separately of course. I don't think that the older builder created special containers for itself? > I'm not quite sure I understand what you are trying to do with nfs there. But you can definitely export the cache to a local filesystem and import it with cache-from. Which is dog-slow, because it…

> I don't think that the older builder created special containers for itself?

Buildkit doesn't create special containers for itself? It's literally a service integrated into dockerd.

> Which can only refer to images that are built inside this particular buildkit or are pullable from a repo.

No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be whatever it wants.

> This is really all I want, a way to quickly reused the previous state saved in some format in Github Cache, NFS, or other storage.

You can cache to GitHub actions cache, S3, az blob, gcs, registries, or export to the client.

Anything you want to stick it on is going to require copying the data, and yeah that's going to be expensive.

Re: A faster path to container images in Bazel

#56

Post by lewo »

I think this is really close to the way nix2container works (https://github.com/nlewo/nix2container). nix2container generates metadata at build time and streams the required data at runtime.

At build time, it generates a JSON file describing the image metadata and the layers data location. At runtime, it consumes this JSON file to stream layer data and image configuration to a destination. This is implemented by adding a new transport to Skopeo. Thanks to this implementation, nix2container doesn't need to handle all various destrination since this is managed by Skopeo itself.

Recently, we introduced a mechanism to also produce such kind of JSON file for the base image (see https://github.com/nlewo/nix2container?tab=readme-ov-file#ni...).

I'm pretty sure the added (not usptreamed yet) transport could be useful in some other close contexts, such as Bazel or Guix.

I'm the nix2container author and i will be glad to discuss with you if you think this Skopeo transport could be useful for you!

(btw, your blog post is pretty well written!)

Re: A faster path to container images in Bazel

#57

Post by cyberax »

cpuguy83 wrote: post #55
cyberax wrote: post #54

Earlier quoted context omitted.

> Buildkit is integrated into dockerd the same way the old builder was. If you want a newer Buildkit you'll need to run it separately of course. I don't think that the older builder created special containers for itself? > I'm not quite sure I understand what you are trying to do with nfs there. But you can definitely export the cache to a local filesystem and import it with cache-from. Which is dog-slow, because it…

> I don't think that the older builder created special containers for itself? Buildkit doesn't create special containers for itself? It's literally a service integrated into dockerd. > Which can only refer to images that are built inside this particular buildkit or are pullable from a repo. No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be what…

> Buildkit doesn't create special containers for itself? It's literally a service integrated into dockerd.

No, it's not. It's a utility container that is hidden from the normal "docker ps". You can see it easily when you use docker-compose with podman.

The easiest way to see it in regular Docker is to create a simple Dockerfile with 'RUN sleep 1000' at the end and start building it. Then enter the Docker host ("docker run -it --rm --privileged --pid=host justincormack/nsenter1") and do 'mount' to see the mounts.

You'll see that buildkit will have its own overlay tree ('/var/lib/docker/buildkit/containerd-overlayfs') and the executor will have its own separate branch too. However, they do share the layers. Now wait for the container to complete building and run it.

You'll see that the running container uses an entirely _different_ set of layers. There is no reuse of layers between the buildkit and the running image.

Yes, the Docker buildkit is technically a daemon that is co-located with dockerd and just runs in its own cgroup tree. But it might as well be remote, because the resulting image runs in a completely different environment.

And the way the image is transferred from buildkit is through the containerd. Which is another separate container in the "moby" namespace.

> No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be whatever it wants.

Any examples?

> You can cache to GitHub actions cache, S3, az blob, gcs, registries, or export to the client.

Go on, try it. Here's a minimal repro: https://gist.github.com/Cyberax/61e6b419cd338ae7c3a7c7098abe...

First, you can build the base image, with the GHA or registry cache. It works. But the `proto` stage will never use cache. The "base" image is supplied through an additional context.

Re: A faster path to container images in Bazel

#58

Post by manmal »

mgaunard wrote: post #50
manmal wrote: post #46

Earlier quoted context omitted.

Is the reason that it works for big tech that those can spare dozens of engineers to make it work?

Big Tech are large organizations with different needs than that of small organizations. They care about setting standards over widely different teams and managing large-scale upgrades etc. It's not optimized for velocity or cost efficiency, which is what a smaller organization needs.

But it doesn’t sound standardized at all if so much customization work is needed.

Re: A faster path to container images in Bazel

#59

Post by cpuguy83 »

cyberax wrote: post #57
cpuguy83 wrote: post #55

Earlier quoted context omitted.

> I don't think that the older builder created special containers for itself? Buildkit doesn't create special containers for itself? It's literally a service integrated into dockerd. > Which can only refer to images that are built inside this particular buildkit or are pullable from a repo. No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be what…

> Buildkit doesn't create special containers for itself? It's literally a service integrated into dockerd. No, it's not. It's a utility container that is hidden from the normal "docker ps". You can see it easily when you use docker-compose with podman. The easiest way to see it in regular Docker is to create a simple Dockerfile with 'RUN sleep 1000' at the end and start building it. Then enter the Docker host ("docke…

If by "utility" container you mean the containers aren't managed under the same stack, that is true. Buildkit, at least prior to docker 29, executes runc directly. It is still using the same storage backend, though there is a shim involved to convert docker's (now deprecated) graph drivers to containerd's snapshotter interface which is what Buildkit speaks. That's why there is a different tree. As of docker 29, containerd's storage is used by default. I can't recall if this used containerd to execute containers or just storage.

> > No, it supports anything Buildkit can fetch: git, http, client dir... for that matter the client itself can shim that to be whatever it wants. > Any examples?

--build-context foo=https://github.com/example/repo.git

Then you can "FROM foo" or whatever you want to do with that context.

> First, you can build the base image, with the GHA or registry cache. It works. But the `proto` stage will never use cache. The "base" image is supplied through an additional context.

What are you expecting to cache here? Are you saying using an extra context like this is preventing it from using the cache?

Re: A faster path to container images in Bazel

#60

Post by hxtk »

forrestthewoods wrote: post #13
dilyevsky wrote: post #10

Earlier quoted context omitted.

> I’m pretty shocked that the Bazel workflow involves downloading Docker base images from external URLs. That seems very unbazel like! That belongs in the monorepo for sure. Not every dependency in Bazel requires you to "first invent the universe" locally. Lots of examples of this like toolchains, git_repository, http_archive rules and on and on. As long as they are checksum'ed (as they are in this case) so that you…

Everything belongs in version control imho. You should be able to clone the repo, yank the network cable, and build. I suppose a URL with checksum is kinda sorta equivalent. But the article adds a bunch of new layers and complexity to avoid “downloading Cuda for the 4th time this week”. A whole lot of problems don’t exist if they binary blobs exist directly in the monorepo and local blob store. It’s hard to describe…

If you did that, Bazel would work a lot better. Most of the complexity of Bazel is because it was originally basically an export of the Google internal project "Blaze," and the roughest pain points in its ergonomics were pulling in external dependencies, because that just wasn't something Google ever did. All their dependencies were vendored into their Google3 source tree.

WORKSPACE files came into being to prevent needing to do that, and now we're on MODULE files instead because they do the same things much more nicely.

That being said, Bazel will absolutely build stuff fully offline if you add the one step of running `bazel sync //...` in between cloning the repo and yanking the cable, with some caveats depending on how your toolchains are set up and of course the possibility that every mirror of your remote dependency has been deleted.

Post reply on HN