Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The next generation of Bazel builds

blogsystem5.substack.com

Post reply on HN
Print view View original

21–30 of 90 posts

Re: The next generation of Bazel builds

#21

Post by jmmv »

mdaniel wrote: post #17

> (I can’t name names because they were never public, but if you probe ChatGPT to see if it knows about these efforts, it somehow knows specific details.) I'm going to laugh if Google joins the NYT in suing OpenAI for copyright infringement from having trained on proprietary data. That said: I also wonder how that actually made it in

These were not well-guarded secrets (and the ideas were pretty "obvious" if you had faced the problems before in dealing with Bazel and large codebases). In talking to people at the conference, several knew about these efforts (and some even knew the codenames), so I assume these had been mentioned here-and-there in discussion threads.

Re: The next generation of Bazel builds

#22

Post by tetromino_ »

throwaway127482 wrote: post #10

One thing not mentioned is the massive amount of complexity and difficulty involved in Starlark rules. In the next generation of build tools, I really wish Starlark could be replaced with some subset of TypeScript, which would have drastically better IDE support. And I wish the typing would be done in such a way that it's harder to do things that are bad for performance. Starlark is super difficult to read, navigate,…

You'll be happy to know that the Starlark team has recently started working on adding static typing! See https://github.com/orgs/bazelbuild/projects/21 and https://bazel.build/rules/language#StarlarkTypes

> a lot of performance gotchas

Can you list some examples?

Re: The next generation of Bazel builds

#23

Post by liminal »

zokier wrote: post #5

I find the dismissal of buck2 pretty shallow. Most of the world is not already heavily invested to bazel, so compatibility is imho overstated; I don't see it being that far-fetched for something like buck2 to leapfrog bazel. That being said, buck2 definitely would need some love from outside meta to really be viable competitor, right now it still feels like half-complete code drop

Buck2 doesn't seem to support JS/TS, so it's not ready for our use case.

Re: The next generation of Bazel builds

#24

Post by tetromino_ »

londons_explore wrote: post #18

I just don't understand how the decision of which bits of a project need rebuilding can be so complex. If I edit 50 lines of code in a 10GB project, then rebuild, the parts that need rebuilding are the parts that read those files when they were last built. So... The decision of what to rebuild should take perhaps a millisecond and certainly doable locally.

50 lines in a single .cc source file which is only compiled once to produce the final artifact - sure, easy to handle.

Now consider that you are editing 50 lines of source for a tool which will then need to be executed on some particular platform to generate some parts of your project.

Now consider that you are editing 50 lines defining the structure and dependency graph of your project itself.

Re: The next generation of Bazel builds

#25

Post by lihaoyi »

A big problem with Bazel not mentioned here is the complexity. It's just really hard for many people to grasp, and adopting Bazel at the two places I worked was a ~10 person-year effort for the rollout with ongoing maintenance after. That's a lot of effort!

IMO Bazel has a lot of good ideas to it: hierarchical graph-based builds, pure hermetic build steps, and so on. Especially at the time, these were novel ideas. But in Bazel they are buried behind a sea of other concepts that may not be so critical: `query` vs `aquery` vs `cquery`, action-graph vs configured-action-graph vs target-graph, providers vs outputs, macro vs rule-impl, etc. Some of these are necessary for ultra-large-scale builds, some are compromises due to legacy, but for the vast majority of non-Google-scale companies there may be a better way.

I'm hoping the next generation of build tools can simplify things enough that you don't need a person-decade of engineering work to adopt it. My own OSS project Mill (https://mill-build.org/) is one attempt in that direction, by re-using ideas from functional and object-oriented programming that people are already familiar with to make build graphs easier to describe and work with. Maybe a new tool won't be able to support million-file monorepos with 10,000 active contributors, but I think for the vast majority of developers and teams that's not a problem at all.

Re: The next generation of Bazel builds

#26

Post by keddad »

IshKebab wrote: post #8
zokier wrote: post #5

I find the dismissal of buck2 pretty shallow. Most of the world is not already heavily invested to bazel, so compatibility is imho overstated; I don't see it being that far-fetched for something like buck2 to leapfrog bazel. That being said, buck2 definitely would need some love from outside meta to really be viable competitor, right now it still feels like half-complete code drop

I agree. The biggest issue with Buck 2 for me (apart from documentation) is the lack of something like bzlmod. There's actually a decent number of modules available for Bazel now: https://registry.bazel.build/all-modules But with Buck2 you're stuck with `http_archive` and vendoring.

Seems like a non-issue to me. I work with a relatively large Bazel monorepo, and we have to vendor pretty much anything anyways. Many 3rd-party rules might need patches to work properly with our custom toolchains/rules.

Bzlmod sounds nice for small projects, but for big monorepos within organizations with established processes it is more of a hassle, and I imagine that most Bazel users are not using it for small projects.

Re: The next generation of Bazel builds

#27

Post by rfoo »

londons_explore wrote: post #18

I just don't understand how the decision of which bits of a project need rebuilding can be so complex. If I edit 50 lines of code in a 10GB project, then rebuild, the parts that need rebuilding are the parts that read those files when they were last built. So... The decision of what to rebuild should take perhaps a millisecond and certainly doable locally.

It is well possible by changing 50 lines of code in a 10GB project you have to rebuild the entire project, if everything (indirectly) depends on what you just changed.

Re: The next generation of Bazel builds

#29

Post by hn_go_brrrrr »

jeffbee wrote: post #19

I find anyone this dismissive of Java hard to take seriously. Java is a cutting-edge language platform where the achievable performance at the limit is on-par with C++. Also, the article's hook didn't grab me. My laptop is way, way more powerful than any "beefy workstation" they ever assigned me at Google, starting with a dual-core 2400MHz Opteron. Bazel's server performance on my laptop or workstation are total non-…

Why do you think this author (a long-time senior Blaze maintainer) believes this? Do you think the Blaze team just not have the necessary expertise?

Re: The next generation of Bazel builds

#30

Post by tetromino_ »

jeffbee wrote: post #19

I find anyone this dismissive of Java hard to take seriously. Java is a cutting-edge language platform where the achievable performance at the limit is on-par with C++. Also, the article's hook didn't grab me. My laptop is way, way more powerful than any "beefy workstation" they ever assigned me at Google, starting with a dual-core 2400MHz Opteron. Bazel's server performance on my laptop or workstation are total non-…

Performance of Java in general and Bazel in particular tends to be not CPU-limited but memory-limited. There was a time when MacBooks maxed out at 16 GB, and running Chrome + an IDE + large Bazel build on a machine with <16 GB memory was not a great experience.
Post reply on HN