Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The next generation of Bazel builds

blogsystem5.substack.com

Post reply on HN
Print view View original

81–90 of 90 posts

Re: The next generation of Bazel builds

#81

Post by aseipp »

IshKebab wrote: post #8
zokier wrote: post #5

I find the dismissal of buck2 pretty shallow. Most of the world is not already heavily invested to bazel, so compatibility is imho overstated; I don't see it being that far-fetched for something like buck2 to leapfrog bazel. That being said, buck2 definitely would need some love from outside meta to really be viable competitor, right now it still feels like half-complete code drop

I agree. The biggest issue with Buck 2 for me (apart from documentation) is the lack of something like bzlmod. There's actually a decent number of modules available for Bazel now: https://registry.bazel.build/all-modules But with Buck2 you're stuck with `http_archive` and vendoring.

To be honest, I'd rather not see Buck2 repeat the mistakes of Bazel so early on, especially when it took a lot of time before settling on bzlmod.

Frankly, I'd rather it go the other way: just have a gigantic 'buck2pkgs' repo that has everything inside of it -- literally fucking everything -- just like Nix and Nixpkgs have done. I've watched and committed to Nixpkgs for over 10 years now and I think being a monorepo has massively contributed to its success, ease of contribution, and general usability. And in practice it means 99% of projects only need one "dependency" and they are done and have everything they could want.

In theory a buck2pkgs could even be completely divorced from Meta's existing Prelude, because you can just reimplement it all yourself, without being constrained by their backwards compatibility with Buck1. That would reduce the surface area between the two projects substantially and let you do a lot of huge cleanups in the APIs.

I actually started doing this a while back but it is of course a massive amount of work. I do think it's a much better way to go, though...

Re: The next generation of Bazel builds

#82

Post by aseipp »

jmmv wrote: post #59
jeffbee wrote: post #33

Earlier quoted context omitted.

I have no idea but dismissing Java as not "a real systems language" just isn't a convincing critique.

As I said in the article, the lack of green threads, lack of value types, and lack of primitive boxing are all things that have gotten in the way of optimizing Bazel to the levels shown by the prototype Go reimplementation. These are all things that you'd expect a systems language to have, and Java did not have them. These are finally materializing now, but it's a little too late. Then you also have the slow startup…

FWIW, the client server design is also used in Buck2, but it has other advantages than just startup time, like keeping track of the filesystem with inotify or watchman so that it already has fresh information about the state of the build graph by the time you run `build`.

Re: The next generation of Bazel builds

#83

Post by fire_lake »

crabbone wrote: post #44
greener_grass wrote: post #7

Earlier quoted context omitted.

Everyone says they want a "tiny" "minimal" "lean" build-system, but there is lots of real complexity in these areas: - Cross-compilation and target platform information - Fetching dependencies - Toolchains I'm not sure a system that solves these would still be considered minimal by most, but those are table-stakes features in my view. If you don't need these things, maybe stick with Make?

I believe that the biggest problem is that different "compilers" do different amount of work. In the race to win the popularity contest many, and especially newer languages offer compilers packaged with "compiler frontend", i.e. a program that discovers dependencies between files, links individual modules into the target programs or libraries, does code generation etc. This prevents creation of universal build system…

I have seen this first hand with Bazel. You have lots of Bazel rules that are partial reimplementations of the language specific tooling. It usually works better - until you hit a feature that isn’t supported.

Re: The next generation of Bazel builds

#84

Post by aseipp »

lostdog wrote: post #55
jeffbee wrote: post #33

Earlier quoted context omitted.

I have no idea but dismissing Java as not "a real systems language" just isn't a convincing critique.

For one, Java programs always take too long to start, and that's not something you want in a program that's supposed to be interactive.

I don't disagree in general but in Bazel's case this path has been heavily optimized. Maybe there are limits to it but "java startup slow" is a 101-level complaint.

In fact, I don't even think the client program for Bazel is written in Java, but C++, and the Java daemon it talks to is started when you first attach to a Bazel workspace and it persists, so subsequent interactions are very fast. Just running `bazel` randomly is not truly indicative of what using it feels like, because that's not what people actually use it for. People use it inside a build workspace, that is the case that matters.

Beyond that, other build systems like Buck2 (Rust) also use the client-daemon architecture for a number of other reasons, including the fact that actually-really-large build graphs are far too large to rebuild on every invocation, so the daemon is necessary to keep the build graph and interactively analyze and incrementally invalidate it on demand. Doesn't matter if it's Rust or Java, these things are architectural. That's also one of the points of the article, of course, and why they're theorizing about server-side analysis caches and graphs, etc.

This all indicates to me that the people designing these systems actually do care about interactivity of their tool. It is not a matter of "java startup slow" to use a client-server design, though it certainly is known to help.

Re: The next generation of Bazel builds

#85

Post by spwa4 »

anon6362 wrote: post #71
sepositus wrote: post #42

Earlier quoted context omitted.

> know it was written by the Very Smart People at google For Google. That's the key. I have the privilege of experiencing both sides, having been at Google for nine years. I never had a problem with Blaze, but using Bazel in a smaller company has been extremely painful. I think there are just very few places that have the exact problems as Google where something like Bazel would be a great fit.

That's the rub. It provides scalability for very large organization, of which, there are few. It's similar to running OpenStack. Meta also has some projects like this, such as buck2 which lacks the really good virtual FS acceleration stuff (eden). Megacorp FOSS tend to skew towards offering whizbang features that are incomplete, complicated, poorly documented, and require a lot of extra work.

Actually if you could make something like github, where all software would be part of a single megarepo and built constantly that would be incredibly useful, and bazel would be excellent for that (or at least the closest thing we have to reasonable)

The problem with bazel and almost every other build system (all except the "scan the source files and build a dependency graph" ones) is that you'll be writing build instructions for all your dependencies that aren't using it. If that was done for you, they'd be incredible.

Re: The next generation of Bazel builds

#86

Post by sunshowers »

joshuamorton wrote: post #79
sunshowers wrote: post #65

Earlier quoted context omitted.

Demonization introduces a range of potential cache invalidation issues. The issues are solvable, but whose KPIs depend on getting to the bottom of them?

Do you have specific examples in the context of blaze/bazel here? I think "the set of cache invalidation issues" you're describing are basically "the set of cache invalidation issues blaze/bazel intends to solve", so the answer to "whose KPIs" is "the blaze team".

I haven't used Bazel but I have used buck1 extensively, and the daemonized mode was quite buggy and often required the process to be killed. Quite frequently the process was just wedged, and even when it wasn't, memory leaks were quite common. Standard black box debuggers like strace and DTrace also become harder to use (e.g. you need to attribute a particular set of syscalls to a client, and a mindless "strace buck build ..." doesn't do what you want).

Daemonization is sometimes necessary, but it introduces lifecycle management problems that get in the way of robustness. Daemonization simply because your choice of programming language has bad startup times doesn't seem like a great idea to me.

I think on-disk cache invalidation and in-memory cache invalidation are distinctly different in practice.

Re: The next generation of Bazel builds

#87

Post by jeffbee »

sunshowers wrote: post #86
joshuamorton wrote: post #79

Earlier quoted context omitted.

Do you have specific examples in the context of blaze/bazel here? I think "the set of cache invalidation issues" you're describing are basically "the set of cache invalidation issues blaze/bazel intends to solve", so the answer to "whose KPIs" is "the blaze team".

I haven't used Bazel but I have used buck1 extensively, and the daemonized mode was quite buggy and often required the process to be killed. Quite frequently the process was just wedged, and even when it wasn't, memory leaks were quite common. Standard black box debuggers like strace and DTrace also become harder to use (e.g. you need to attribute a particular set of syscalls to a client, and a mindless "strace buck…

> I haven't used Bazel

That was the perfect place to end the comment.

Re: The next generation of Bazel builds

#88

Post by joshuamorton »

sunshowers wrote: post #86
joshuamorton wrote: post #79

Earlier quoted context omitted.

Do you have specific examples in the context of blaze/bazel here? I think "the set of cache invalidation issues" you're describing are basically "the set of cache invalidation issues blaze/bazel intends to solve", so the answer to "whose KPIs" is "the blaze team".

I haven't used Bazel but I have used buck1 extensively, and the daemonized mode was quite buggy and often required the process to be killed. Quite frequently the process was just wedged, and even when it wasn't, memory leaks were quite common. Standard black box debuggers like strace and DTrace also become harder to use (e.g. you need to attribute a particular set of syscalls to a client, and a mindless "strace buck…

I asked this because I've used bazel some, and blaze fairly extensively, and despite having done some deeply cursed things to blaze at times, I've never had issues with it as a daemon, to the point where I'd never consider needing to run it in non-daemonized mode as part of a debug process. It just works.

Second, "startup-time" is probably the least relevant complaint here in terms of why to daemonize. Running `help` without a daemon set up does take a few seconds (and yeah that's gross), but constructing the action graph cold for $arbitrary_expensive_thing takes a minute or two the first time I do it, and then around 1 second the next time.

Caching the build graph across builds is valuable, and persisting it in memory makes a lot more sense than on disk, in this case. The article we're discussing even argues in favor of making this even more extreme and moving action graph calculation entirely into a service on another machine because it prevents the situation where the daemon dies and you lose your (extremely valuable!) cached analysis graph:

> Bonanza performs analysis remotely. When traditional Bazel is configured to execute all actions remotely, the Bazel server process is essentially a driver that constructs and walks a graph of nodes. This in-memory graph is known as Skyframe and is used to represent and execute a Bazel build. Bonanza lifts the same graph theory from the Bazel server process, puts it into a remote cluster, and relies on a distributed persistent cache to store the graph’s nodes. The consequence of storing the graph in a distributed storage system is that, all of a sudden, all builds become incremental. There is no more “cold build” effect like the one you see with Bazel when you lose the analysis cache.

If you're worried about cache invalidation and correctness issues due to daemonization, I think you'd want to be even more concerned about moving them entirely off machine.

(I'm also not sure how their proposal intends to manage e.g. me and Phil on another machine trying to build conflicting changes that both significantly impact how the analysis graph is calculated and thrash each other, either you have to namespace so they don't thrash and then you've moved the daemon to the cloud, or you do some very very fancy partial graph invalidation approach, but that isn't discussed and it feels like it would be white paper worthy)

Re: The next generation of Bazel builds

#89

Post by sunshowers »

jeffbee wrote: post #87
sunshowers wrote: post #86

Earlier quoted context omitted.

I haven't used Bazel but I have used buck1 extensively, and the daemonized mode was quite buggy and often required the process to be killed. Quite frequently the process was just wedged, and even when it wasn't, memory leaks were quite common. Standard black box debuggers like strace and DTrace also become harder to use (e.g. you need to attribute a particular set of syscalls to a client, and a mindless "strace buck…

> I haven't used Bazel That was the perfect place to end the comment.

Are you saying that daemonization doesn't make black box debugging harder?

Re: The next generation of Bazel builds

#90

Post by sunshowers »

joshuamorton wrote: post #88
sunshowers wrote: post #86

Earlier quoted context omitted.

I haven't used Bazel but I have used buck1 extensively, and the daemonized mode was quite buggy and often required the process to be killed. Quite frequently the process was just wedged, and even when it wasn't, memory leaks were quite common. Standard black box debuggers like strace and DTrace also become harder to use (e.g. you need to attribute a particular set of syscalls to a client, and a mindless "strace buck…

I asked this because I've used bazel some, and blaze fairly extensively, and despite having done some deeply cursed things to blaze at times, I've never had issues with it as a daemon, to the point where I'd never consider needing to run it in non-daemonized mode as part of a debug process. It just works. Second, "startup-time" is probably the least relevant complaint here in terms of why to daemonize. Running `help`…

Yeah, moving more things into the cloud makes debuggability even worse than today. I think that's a very large downside.
Post reply on HN