Live data from Hacker News

The next generation of Bazel builds

blogsystem5.substack.com

41–50 of 90 posts

Re: The next generation of Bazel builds

#41
post #25

A big problem with Bazel not mentioned here is the complexity. It's just really hard for many people to grasp, and adopting Bazel at the two places I worked was a ~10 person-year effort for the rollout with ongoing maintenance after. That's a lot of effort! IMO Bazel has a lot of good ideas to it: hierarchical graph-based builds, pure hermetic build steps, and so on. Especially at the time, these were novel ideas. Bu…

> other concepts that may not be so critical: `query` vs `aquery` vs `cquery`, action-graph vs configured-action-graph vs target-graph, providers vs outputs, macro vs rule-impl, etc

Almost all of the distinctions you mentioned are related to the way that Bazel has the concept of a "target", which lets the build graph work at a higher level than individual files.

Suppose you write the following in a BUILD file:

    cc_library(
        name = "foo",
        srcs = ["foo.c"],
        hdrs = ["foo.h"],
    )

    cc_library(
        name = "bar",
        srcs = ["bar.c"],
        hdrs = ["bar.h"],
        deps = [":foo"],
    )
This lets us define, at a high level, that ":foo" and ":bar" are C/C++ libraries, and that bar depends on foo. This is the build graph of targets, and it's independent of any particular files that these rules may produce (.o, .a, .so, etc).

It's nice to be able to query the build graph at this high level. It lets you see the relationship between components in the abstract, rather than a file-by-file level. That is what "bazel query" does.

But sometimes you might want to dig deeper into the specific commands (actions) that will be executed when you build a target. That is what "bazel aquery" is for.

Macros vs. rules is basically a question of whether the build logic runs before or after the target graph is built. A macro lets you declare a bit of logic where something that looks like a target will actually expand into multiple targets (or have the attributes munged a bit). It is expanded before the target graph is built, so you won't see it in the output of "bazel query."

If you took away the target graph, I think you'd take away a lot of what makes Bazel powerful. A key idea behind Bazel is to encapsulate build logic, so that you can use a rule like cc_library() without having to know how it's implemented or exactly what actions will run.

I don't say this to minimize any of the pain people experience when adopting Bazel. I'm actually curious to learn more about what the biggest pain points are that make it difficult to adopt.

Re: The next generation of Bazel builds

#42
post #37
post #25

A big problem with Bazel not mentioned here is the complexity. It's just really hard for many people to grasp, and adopting Bazel at the two places I worked was a ~10 person-year effort for the rollout with ongoing maintenance after. That's a lot of effort! IMO Bazel has a lot of good ideas to it: hierarchical graph-based builds, pure hermetic build steps, and so on. Especially at the time, these were novel ideas. Bu…

I'm glad to see someone else describe their experience this way too. bazel has arrived at $WORK and it has been a non-trivial amount of work for even the passionate advocates of bazel. I know it was written by the Very Smart People at google. They are clearly smarter than me so I must be the dummy. Especially since I never passed their interview tests. :-) Of course given all things google, by the time I'm fully onbo…

> know it was written by the Very Smart People at google

For Google. That's the key. I have the privilege of experiencing both sides, having been at Google for nine years. I never had a problem with Blaze, but using Bazel in a smaller company has been extremely painful. I think there are just very few places that have the exact problems as Google where something like Bazel would be a great fit.

Re: The next generation of Bazel builds

#43

I just don't understand how the decision of which bits of a project need rebuilding can be so complex. If I edit 50 lines of code in a 10GB project, then rebuild, the parts that need rebuilding are the parts that read those files when they were last built. So... The decision of what to rebuild should take perhaps a millisecond and certainly doable locally.

I mean, easy things are easy to build?

It is not at all uncommon to have changes percolate out into larger impacts than you'd expect, though. Especially in projects that attempt whole program optimizations as part of the build.

Consider anything that basically builds a program that is used at build time. Which is not that uncommon when you consider that ML models have grown significantly. Change that tool, and suddenly you have to rebuild the entire project if you didn't split it out into a separate graph. (I say ML, but really any simple linter/whatever is the same here.)

Re: The next generation of Bazel builds

#44
post #4

>On the other hand, we need a tiny build system that does all of the work locally and that can be used by the myriad of open-source projects that the industry relies on. This system has to be written in Rust (oops, I said it) with minimal dependencies and be kept lean and fast so that IDEs can communicate with it quickly. This is a niche that is not fulfilled by anyone right now and that my mind keeps coming to; Yes…

Everyone says they want a "tiny" "minimal" "lean" build-system, but there is lots of real complexity in these areas: - Cross-compilation and target platform information - Fetching dependencies - Toolchains I'm not sure a system that solves these would still be considered minimal by most, but those are table-stakes features in my view. If you don't need these things, maybe stick with Make?

I believe that the biggest problem is that different "compilers" do different amount of work. In the race to win the popularity contest many, and especially newer languages offer compilers packaged with "compiler frontend", i.e. a program that discovers dependencies between files, links individual modules into the target programs or libraries, does code generation etc. This prevents creation of universal build systems.

I.e. javac can be fed inputs of individual Java source files, similar to GCC suite compilers, but Go compiler needs a configuration for the program or the library it compiles. Then there are also systems like Cargo (in Rust) that also do part of the job that the build system has to do for other languages.

From a perspective of someone who'd like to write a more universal build system, encountering stuff like Cargo is extremely disappointing: you immediately realize that you will have to either replace Cargo (and nobody will use your system because Cargo is already the most popular tool and covers the basic needs of many simple projects), or you will have to add a lot of work-arounds and integrations specific to Cargo, depend on their release cycle, patch bugs in someone else's code...

And it's very unfortunate because none of these "compiler frontends" come with support for other languages, CI, testing etc. So, eventually, you will need an extra tool, but by that time the tool that helped you to get by so far will become your worst enemy.

Re: The next generation of Bazel builds

#45

I just don't understand how the decision of which bits of a project need rebuilding can be so complex. If I edit 50 lines of code in a 10GB project, then rebuild, the parts that need rebuilding are the parts that read those files when they were last built. So... The decision of what to rebuild should take perhaps a millisecond and certainly doable locally.

> the parts that need rebuilding are the parts that read those files when they were last built...

...and the transitive closure of those parts, which is where things get complicated. It may be that the output didn't actually change, so you can prune the graph there with some smarts. It may be that the thing changed was a tool used in many other rules.

And you have to know the complete set of outputs and inputs of every action.

And on and on.

Re: The next generation of Bazel builds

#46
Great article, that gets the critique exactly right. The most frustrating part of Bazel is how shoddy the workmanship is. For example, Bazel throws away your analysis cache when you change flags that have nothing to do with what's being built or how, like flags that change what tests are run.

If course the biggest issue is that tiny operations take more time than necessary. For example, at $PreviousJob we wrote custom CLI tools and put them in our monorepo, but since you need Bazel to run them, running a basic tool took 9-12 seconds. So this revelation from the article was totally unsurprising:

> This rewrite was carefully crafted to optimize memory layouts, avoiding unnecessary pointer chasing (which was impossible to avoid in Java). The results of this experiment proved that Blaze could be made to analyze large portions of Google’s build graph in just a fraction of the time, without any sort of analysis caching or significant startup penalties.

Yeah, with attentive engineering it's not surprising that you can get massive speedups.

Finally, Bazel is ok at building multiple languages, but awful at getting them to depend on each other. I don't know what's going on here, but whatever magic makes it possible for any language to depend on C and C++ was not extended to other possible dependencies. So now you get to fight or rewrite all the rules, which is another bag of worms.

Re: The next generation of Bazel builds

#47
post #5

I find the dismissal of buck2 pretty shallow. Most of the world is not already heavily invested to bazel, so compatibility is imho overstated; I don't see it being that far-fetched for something like buck2 to leapfrog bazel. That being said, buck2 definitely would need some love from outside meta to really be viable competitor, right now it still feels like half-complete code drop

The biggest problem with everything NotBazel in this space is IDE support. JetBrains have already moved their build for IntelliJ to it, and are aggressively adding native first party support.

Re: The next generation of Bazel builds

#48

I just don't understand how the decision of which bits of a project need rebuilding can be so complex. If I edit 50 lines of code in a 10GB project, then rebuild, the parts that need rebuilding are the parts that read those files when they were last built. So... The decision of what to rebuild should take perhaps a millisecond and certainly doable locally.

That rule misses important cases:

• Adding a file. It hasn't been read before, so no tasks in your graph know about it. If you can intercept and record what file patterns a build tool is looking for it helps, but you can't easily know that because programs often do matching against directory contents themselves, not in a way you can intercept.

• File changes that yield no-op changes, e.g. editing a comment in a core utility shouldn't recompile the entire project. More subtly, editing method bodies in a Java program doesn't require the users to be recompiled, but editing class definitions or exposed method prototypes does.

• "Building" test cases.

• You don't want to repeat work that has been done before, so you want to cache it (e.g. switching branches back and forth shouldn't rebuild everything).

Re: The next generation of Bazel builds

#49
In the 2020s I have used BUCK, BUCK2, and Blaze every working day. I don’t feel like getting in to details but I’ll just say that BUCK2 is the best of the 3 and I’m delighted it’s open source.

I don’t have experience with Bazel as it exists externally but I expect the internal Blaze has advanced far beyond it by now as the entire Google engineering world depends on it.

Re: The next generation of Bazel builds

#50
post #17

> (I can’t name names because they were never public, but if you probe ChatGPT to see if it knows about these efforts, it somehow knows specific details.) I'm going to laugh if Google joins the NYT in suing OpenAI for copyright infringement from having trained on proprietary data. That said: I also wonder how that actually made it in

I can say with full certainty that thousands of engineers at Google were feeding their day to day LLM assisted coding queries in to the ChatGPT web ui for about 2 years before a custom Chrome extension was pushed to employee laptops to stop it.

So GPT has likely post-trained on a lot of Google code knowledge.

Post reply on HN