Scaling Rust Builds with Bazel • Hacker News Board

Re: Scaling Rust Builds with Bazel

#51

Post by echelon » Fri, Mar 24, 2023, 1:25 PM UTC

habitue wrote: post #12

Hey, what a surprise, ex-google engineers dragging everyone into bazel... Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. There are languages that need bazel, rust isn't one of them

> There are languages that need bazel, rust isn't one of them

Speak for yourself. Our Rust monorepo has nearly a dozen apps and counting, lots of shared libraries, C++ dependencies, etc. Tests require multiple cargo invocations.

We need a proper build system with distributed build caching. Cargo takes forever and doesn't understand the workloads.

Re: Scaling Rust Builds with Bazel

#52

Post by Game_Ender » Fri, Mar 24, 2023, 1:25 PM UTC

reitanuki wrote: post #49

bjackman wrote: post #22

Earlier quoted context omitted.

Lol, Googler here, I can't deny that I have he urge to force Bazel into the world. It's just, when I joined Google I found a build system I didn't hate. This was the first time. I don't actually love Bazel (Blaze internally), but I love that I don't hate it. Every time I have to learn a new build system in open source work I groan. Why are there so many systems for essentially the same thing?

on the other hand, there are some of us that really dislike Bazel. Try compiling something for ARM and then realising you can't find a build of the build tool for ARM... That was a special level of pain. Tools commonly available in distros are typically much easier to use for outsiders.

FWIW Bazel now ships linux (and Mac and Windows) arm64 binaries: https://github.com/bazelbuild/bazel/releases/download/6.1.1/...

Definitely a ton of rough edges on Bazel but the passage of time is rounding them, even if progress uneven and frustratingly slow.

Re: Scaling Rust Builds with Bazel

#53

Post by JeremyBanks » Fri, Mar 24, 2023, 1:28 PM UTC

illiarian wrote: post #18

keddad wrote: post #9

Earlier quoted context omitted.

You basically have to declare all those dependencies in Bazel. For instance, if you look at rules for Go, they declare the whole dependency graph (Modules, their relations) in Bazel. There is tooling to generate that for you, so it's not really that bad. Where it gets harder is when this language feature is integrated with an existing build system. C# is normally build with it's own build system, and to build it with…

> There is tooling to generate that for you, so it's not really that bad. So you need several layers of tooling just to get something built. It's bad

[dead]

Re: Scaling Rust Builds with Bazel

#54

Post by Game_Ender » Fri, Mar 24, 2023, 1:32 PM UTC

carlmr wrote: post #23

bjackman wrote: post #22

Earlier quoted context omitted.

Lol, Googler here, I can't deny that I have he urge to force Bazel into the world. It's just, when I joined Google I found a build system I didn't hate. This was the first time. I don't actually love Bazel (Blaze internally), but I love that I don't hate it. Every time I have to learn a new build system in open source work I groan. Why are there so many systems for essentially the same thing?

Question for an insider. Bazel code coverage seems to be one of the roughest edges. At least when running C++ tests. Only lcov is supported, and due to the sandboxing it's impossible to get other tools working since they always spit out extra instrumentation data that's neither cached nor available on the next sandbox. Does blaze support code coverage properly? Does Google just not care about code coverage? Am I just…

You can get coverage working for almost anything by having each test convert its coverage data to LCOV before stopping, in the same sandbox as the test.

I forget the exact details but I think you can do this with a custom run_under test wrapper used during coverage. The wrapper runs your test, with whatever custom coverage system you have, then afterwards it converts to LCOV. Sometimes this might require updates to language specific rules because the test runner itself needs to be configured.

The bottom line is that Bazel upstream does not put a lot of attention into this, so it’s under developed outside of Google, because Google internal does not use the OSS rules.

Re: Scaling Rust Builds with Bazel

#55

Post by substation13 » Fri, Mar 24, 2023, 1:34 PM UTC

DC-3 wrote: post #35

keddad wrote: post #32

Earlier quoted context omitted.

There are open-source rules for that, but at my place of work, we found those lacking. We had to write our own set of rules which would basically take all the source files for a given target, generate MSbuild files depending on the target type and call it from Bazel to build the code in this isolated environment. Another problem are nugets: you have to wrap each in a Bazel target in order to make it available during…

That's what I feared. Have you got any solution to make this play nicely with Visual Studio?

You will need to maintain MS Build files alongside the Bazel definitions. In theory the MS Build files can be generated from the Bazel definitions but someone has to write that tool.

Re: Scaling Rust Builds with Bazel

#56

Post by wongarsu » Fri, Mar 24, 2023, 1:38 PM UTC

I can get behind a story of starting with cargo, trying to improve by adding sccache, and when that's not enough switching to Bazel. I believe 99% of projects/organisations are best served with the first two of these steps, but it's nice to have the option of going to Bazel for the other 1% of cases (like when you have a lot of inhouse C++ in the mix).

Re: Scaling Rust Builds with Bazel

#57

Post by WastingMyTime89 » Fri, Mar 24, 2023, 1:54 PM UTC

habitue wrote: post #12

Hey, what a surprise, ex-google engineers dragging everyone into bazel... Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. There are languages that need bazel, rust isn't one of them

Then again, the main issue of Bazel is that it’s complex to setup. If you already have the know how internally, it’s a reasonably good and well supported build system. It’s definitely better than cargo when it comes to caching and the breadth of targets available. Plus cargo is strictly limited to rust.

Re: Scaling Rust Builds with Bazel

#58

Post by duped » Fri, Mar 24, 2023, 2:21 PM UTC

bilekas wrote: post #20

habitue wrote: post #12

Hey, what a surprise, ex-google engineers dragging everyone into bazel... Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. There are languages that need bazel, rust isn't one of them

I'm inclined to agree, but mainly because I can't see an issue where cargo is insufficient, it seems to me that would be an issue somewhere else. Not cargo specific.

Cargo is sufficient if what you're building is an executable or a Rust library.

It still doesn't support post build steps, and integrates poorly with other tools that are used for more general builds.

Re: Scaling Rust Builds with Bazel

#59

Post by zaphar » Fri, Mar 24, 2023, 2:29 PM UTC

illiarian wrote: post #18

keddad wrote: post #9

Earlier quoted context omitted.

You basically have to declare all those dependencies in Bazel. For instance, if you look at rules for Go, they declare the whole dependency graph (Modules, their relations) in Bazel. There is tooling to generate that for you, so it's not really that bad. Where it gets harder is when this language feature is integrated with an existing build system. C# is normally build with it's own build system, and to build it with…

> There is tooling to generate that for you, so it's not really that bad. So you need several layers of tooling just to get something built. It's bad

If you only ever need to build Rust code then cargo is all you should need. However if you need to build Rust, C, C++, python, and JavaScript all together in one build then Cargo is inevitably going to make you sad. That is when you reach for something like Bazel.

Re: Scaling Rust Builds with Bazel

#60

Post by insanitybit » Fri, Mar 24, 2023, 2:48 PM UTC

habitue wrote: post #12

Hey, what a surprise, ex-google engineers dragging everyone into bazel... Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. There are languages that need bazel, rust isn't one of them

> Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything.

I feel like our use case at Grapl wasn't particularly crazy. It was just slightly more than "I'm a dev building a library and publishing it" - we had a workspace with multiple devs and CI/CD. CI/CD was really slow - issues with caching, rerunning unnecessary tests, etc.

Plus we had some Python code so unifying our builds would have been ncie.