Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A faster path to container images in Bazel

tweag.io

Post reply on HN
Print view View original

41–50 of 60 posts

Re: A faster path to container images in Bazel

#41

Post by rienbdj »

gugagore wrote: post #26
mgaunard wrote: post #19

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top. I'm not too surprised that out of the box docker images exhibit more of this. W…

For some more depth into the "bloat" of bazel, I like this reference: https://fzakaria.com/2024/02/27/hermetic-but-at-what-cost

Curious if this bloat is present in a Build Without Bytes scenario?

Re: A faster path to container images in Bazel

#42

Post by __turbobrew__ »

paulddraper wrote: post #22

This is smart. Container layers are so large that moving them around is heavy. So defer that part for the non-hermetic push/load parts of the process, while retaining heremticity/reproducibility. You can sort of think of it like the IO monad in Haskell…defer it all until the impure end.

Is load not hermetic? Ideally you should be mirroring all layers you use as inputs to your OCI builds and pin SHA256 versions. Your caching will also have issues if you don’t pin versions. Push should also be idempotent, but not hermetic.

Re: A faster path to container images in Bazel

#43

Post by __turbobrew__ »

tkinz27 wrote: post #16
jeffbee wrote: post #4

Funny that the article only obliquely references the compression issues. The OCI users that I have seen are using gzip due to inertia, while zstd layers have been supported for a while and are radically faster.

I looked into switching to zstd recently however at least crane the utility that rules_oci uses to upload containers does not yet support uploading zstd layers. https://github.com/google/go-containerregistry/pull/1827

Issue is over two years old. Man it is so sad how hard it can be to upstream work to big open source projects. I have a number of PRs open on both the kubernetes and etcd projects and it is almost impossible to get anyone to review them, and since nobody will review my PRs I cannot get enough work under my belt to be a committer. Sometimes I feel like if you don’t have an @redhat or @google account people just ignore you.

Re: A faster path to container images in Bazel

#44

Post by cpuguy83 »

cyberax wrote: post #27
cpuguy83 wrote: post #24

Earlier quoted context omitted.

Buildkit didn't break anything here except that it each individual build step is no longer exposed as a runnable image in docker. That was unfortunate, but you can actually have buildkit run a command in that filesystem these days, and buildx now even exposes a DAP interface. Buldkit is far more efficient than the old model.

Buildkit is still a separate system, unlike the old builder. So you get that extra step of importing the result back. And since it's a separate system, there are also these strange limitations. For example, I can't just cache pre-built images in an NFS directory and then just push them into the Buildkit context. There's simply no command for it. Buildkit can only pull them from a registry. > Buldkit is far more effic…

Buildkit is integrated into dockerd the same way the old builder was. If you want a newer Buildkit you'll need to run it separately of course.

I'm not quite sure I understand what you are trying to do with nfs there. But you can definitely export the cache to a local filesystem and import it with cache-from. You can also provide named contexts.

"Buildkit can only pull them from a registry" is just plain false.

Re: A faster path to container images in Bazel

#45

Post by ttiurani »

mgaunard wrote: post #19

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top. I'm not too surprised that out of the box docker images exhibit more of this. W…

Seconded. I tried hard to use Bazel in a polyglot repo because I really wanted just one builder.

Unfortunately, the amount of work you need to just maintain the build across language and bazel version upgrades is incredibly high. Let alone adding new build steps, or going even slightly off the well-trodded path.

I feel like Bazel would need at least 5 more full-time engineers to eventually turn it into an actually usable build tool outside Big Tech. Right now many critical open source Bazel rules get a random PR every now and then from people who don't actually (have time to) care about the open source community.

My go-to now is to use mise + just to glue together build artifacts from every language's standard build tools. It's not great but at least I get to spend time on programming instead of fixing the build.

Re: A faster path to container images in Bazel

#46

Post by manmal »

ttiurani wrote: post #45
mgaunard wrote: post #19

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top. I'm not too surprised that out of the box docker images exhibit more of this. W…

Seconded. I tried hard to use Bazel in a polyglot repo because I really wanted just one builder. Unfortunately, the amount of work you need to just maintain the build across language and bazel version upgrades is incredibly high. Let alone adding new build steps, or going even slightly off the well-trodded path. I feel like Bazel would need at least 5 more full-time engineers to eventually turn it into an actually us…

Is the reason that it works for big tech that those can spare dozens of engineers to make it work?

Re: A faster path to container images in Bazel

#47

Post by sluongng »

The underlying problem is that most container images are not cache efficient. Compressed tarballs arent and that’s what most of container images are. And Bazel relies heavily on caching to stay fast.

Most of the hyper scaler actually do not store container images as tarballs at scale. They usually flatten the layers and either cache the entire file system merkle tree, or breaking it down to even smaller blocks to cache them efficiently. See Alibaba Firefly Nydus, AWS Firecracker, etc… There is also various different forms of snapshotters that can lazily materialize the layers like estargz, soci, nix, etc… but none of them are widely adopted.

Re: A faster path to container images in Bazel

#48

Post by yupsurprise »

manmal wrote: post #46
ttiurani wrote: post #45

Earlier quoted context omitted.

Seconded. I tried hard to use Bazel in a polyglot repo because I really wanted just one builder. Unfortunately, the amount of work you need to just maintain the build across language and bazel version upgrades is incredibly high. Let alone adding new build steps, or going even slightly off the well-trodded path. I feel like Bazel would need at least 5 more full-time engineers to eventually turn it into an actually us…

Is the reason that it works for big tech that those can spare dozens of engineers to make it work?

Yes, each of the big techs has teams that just work on the build systems, however it should also be noted that none of the big tech use the open source Bazel, Google uses Blaze internally which is what Bazel is derived from, Amazon uses Brazil which has nothing to do with Bazel and Meta uses Buck, which I know nothing of so I won't comment on it.

The major issue I found when trying to use Bazel was that its essentially a build system without specific rules for each language, hence rules support for each specific language is dependant on each language's specific community, most of which are quite tiny, and mostly maintained by upstreaming changes from their individual companies, servicing their own needs, hence a lot of work is required to make it work for your own company's needs.

Re: A faster path to container images in Bazel

#49

Post by mgaunard »

rienbdj wrote: post #40
mgaunard wrote: post #19

My experience is that anything involving Bazel is slow, bloated, and complicated, hammers your disk, copies your files ten times over, and balloons your disk usage without ever collecting the garbage. A lot of essential features are missing so you realistically have to build a lot of custom rules if not outright additional tooling on top. I'm not too surprised that out of the box docker images exhibit more of this. W…

What can used instead for a large multilanguage repo where we want to build every commit? Genuine question - also find Bazel frustrating at times.

Personally, I write my own build systems.

Any readily available build system is more of a meta-language onto which you code your own logic, but with limited control and capabilities. Might as well take control of the whole stack in a real programming language.

Building my own build system lets me optimize my workflow end-to-end, from modular version management, packaging and releasing, building and testing, tightly integrating whatever tool or reporting I want, all seamlessly under the same umbrella.

I mostly do C++, Assembly, eBPF, Python (including C++ Python modules), and multi-stage codegen on Linux, so I haven't really looked at the complexity of other languages or platforms.

Re: A faster path to container images in Bazel

#50

Post by mgaunard »

manmal wrote: post #46
ttiurani wrote: post #45

Earlier quoted context omitted.

Seconded. I tried hard to use Bazel in a polyglot repo because I really wanted just one builder. Unfortunately, the amount of work you need to just maintain the build across language and bazel version upgrades is incredibly high. Let alone adding new build steps, or going even slightly off the well-trodded path. I feel like Bazel would need at least 5 more full-time engineers to eventually turn it into an actually us…

Is the reason that it works for big tech that those can spare dozens of engineers to make it work?

Big Tech are large organizations with different needs than that of small organizations.

They care about setting standards over widely different teams and managing large-scale upgrades etc.

It's not optimized for velocity or cost efficiency, which is what a smaller organization needs.

Post reply on HN