Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Rust Builds with Bazel

mmapped.blog

Post reply on HN
Print view View original

111–120 of 133 posts

Re: Scaling Rust Builds with Bazel

#111

Post by habitue »

malkia wrote: post #81
tragomaskhalos wrote: post #45

There is a broad tendency in software to jump onto whatever the big FAANG guys are doing. The trouble with this is often that they are working at a vastly bigger scale than you are, hence their solutions are vastly overfitted to your problem. So it's definitely interesting to read about this stuff but I'm sure Cargo is fine for most folks.

There is also big tendendcy to ignore them "because we are not at that scale" without realizing that you've reached the scale where you need it, even though your scale is magnitudes still less than what FAANG is. The fallacy here is that just because you are not at the scale of the FAANG does not mean that their tools/libraries/products are not suitable for you - e.g. it's not the case that only if you reach their sc…

This is true, and so you have to look at it case by case. I think if most developers look at the switching costs of bazel vs. just using a decent build tool like cargo, they wont make that tradeoff.

Similarly, they might choose some other tool out of google because it doesnt extract such a high price

Re: Scaling Rust Builds with Bazel

#112

Post by ye-olde-sysrq »

chatmasta wrote: post #110
kubota wrote: post #64

Earlier quoted context omitted.

Bazel isn't about a particular language needing its build facilities - its about ALL of your organizations languages being built with one tool, ideally in one monorepository. Its very attractive when you have lots of microservices that would otherwise live in disparate repositories. Using a monorepo and Bazel allows you to share api definitions (proto files) across your entire code base (rules_proto, rules_grpc) , so…

My problem with Bazel - at least based on my understanding when I evaluated it - is that you can't use the language-specific package managers for each module in your monorepo, so it's all or nothing. I'd prefer a tool that makes it much easier to defer build tasks to my package manager of choice in each project (eg Yarn for TS, Poetry for Python, etc.) and that lets me define rules at one level of abstraction above t…

This was my main gripe with bazel. It's the worst build tool for any individual language but the best (only?) that can do all of them.

Other big complaints were:

- big CPU hog

- bad IDE integration (esp scala/java for intelliJ)

- it lets people write i-cant-believe-its-not-python, which is far too much rope to hang yourself with. I've seen horrors.

Re: Scaling Rust Builds with Bazel

#113

Post by HervalFreire »

habitue wrote: post #108
HervalFreire wrote: post #78

Earlier quoted context omitted.

Not just that it doesn't integrate well with external tools. Cmake works better here. We had a googler at our company switch everything to bazel and it just slowed everyone down. There really is something about Google engineers, this sort of subtle arrogance that the Google way is superior to everything else is existence. Maybe I'm the ass hole but I wonder if anyone can relate?

I don't think it's arrogance, it's just that blaze inside google is really nice, and they assume that using the tool outside google will also be really nice. The issue is that to make it really nice: - you have to convert everything to it, not just almost everything - you need to set up build servers and lightning fast caching servers - you need a team dedicated to building bazel rules for whatever doesn't already ha…

>- you need a team dedicated to building bazel rules for whatever doesn't already have rules (see: you must convert everything). This team is a bottleneck

This is a major issue. A good build tool shouldn't require a dedicated team.

Ideally a build tool should function as sort of a settings menu for the project.

Re: Scaling Rust Builds with Bazel

#114

Post by pie_flavor »

simplotek wrote: post #103
lern_too_spel wrote: post #61

Earlier quoted context omitted.

In addition to the problems stated in the article that you're ignoring here, in any large enough system, you are going to use multiple languages. Cargo isn't built for that. Bazel is.

> In addition to the problems stated in the article that you're ignoring here What problems, exactly? The author acknowledged cargo works but hand-waves over vague claims of "it does not track dependencies well or support arbitrary build graphs", followed by acknowledging those issues are either non-issues or solved problems whose solution was dismissed without any good reason (caching tools). The author's claims boi…

The author gave a concrete example of such a build graph and a concrete example of why caching tools can't be used effectively.

Re: Scaling Rust Builds with Bazel

#115

Post by pie_flavor »

illiarian wrote: post #6

One question I have after all these articles and internal company presentations is: how exactly does Bazel track dependencies? Module A importing module B importing module C is a language-level feature, not a build system-level feature.

It's in the language and it's in the manifest. The manifest usually ends in `.toml`, but with Bazel it ends in `.bazel`.

Re: Scaling Rust Builds with Bazel

#116

Post by baby »

0x457 wrote: post #106
baby wrote: post #69

Earlier quoted context omitted.

And they’re all horrible

Why? As long as it can take `Cargo.lock` as input instead of making you calculate hash yourself - it's all good. I'm using `crane` and `buildRustPackage` to build wasm front-end that is embedded into server binary (also rust). It works like a charm.

So one problem we have at the moment in our project is that outdated Cargo.lock will basically break everything. Using `nix develop` will just fail to build the derivation, instead of regenerating the correct Cargo.lock (because it depends on someone not using `nix develop` to build the Rust project).

It seems like the ideal way to mix nix and cargo is to simply trust and delegate to cargo once you reach a rust dependency, not try to nixify it.

I think there was a rustup library for nix that probably would allow doing that but I haven't spent enough time trying (it's such a pain to debug flake.nix files)

Re: Scaling Rust Builds with Bazel

#117

Post by kibwen »

yosefk wrote: post #29
habitue wrote: post #12

Hey, what a surprise, ex-google engineers dragging everyone into bazel... Maybe you have some really complicated use case that makes cargo insufficient, but really, it's pretty damn nice and it's integrated with everything. There are languages that need bazel, rust isn't one of them

I never worked for Google nor worked with anyone from Google who said something like "you should move to Bazel" even once. I also did lots of work on in house build system migrations and optimizations. Do you think that the problems with cargo mentioned in TFA are exaggerated or solvable? I think Bazel's support for caching, parallelism and custom rules are real advantages for large projects. What's wrong with this a…

> Do you think that the problems with cargo mentioned in TFA are exaggerated or solvable?

I'd say they're exaggerated at least slightly, yes.

It starts out by saying "Cargo isn't a build system" (which is already the sort of provocative exaggeration designed to increase participation), and then immediately veers off and starts talking about running tests. Cargo is a build system and dependency manager. It also happens to provide `cargo test` for running unit tests and integration tests, but this support is relatively simple, and if you need to do something weird then you're welcome to write a script to serve as your test runner. But a build system is not a test runner; a build system produces artifacts, and a test runner runs tests. The author appears to be annoyed that Cargo is not a build system for web assembly, which is part of their test process, and obviously Cargo is not. This is a somewhat eye-rolling critique, and does little to motivate rewriting the world in Bazel.

The second complaint is that Cargo uses file timestamps to decide when to recompile files. I suppose it is possible that Cargo could then proceed to lex any changed file to produce a comment-free token stream, hash the token stream, and then compare that hash to a previously-computed hash that you've stored in a database somewhere. Frankly, this sounds like a lot of work for essentially zero benefit; how often, exactly, do you go into a file and change only a comment? This is a weak justification.

It then mentions how changing git branches can invalidate the cache, but I don't think that's true. Git only updates timestamps on files that have changed when changing branches. If a file has changed when changing a branch, that is, again, probably because it has actual changes, and will almost never be because only a comment has changed.

It then makes some suggestions about Cargo not properly tracking changes to some global system resource that they have, which is simply too vague to comment on, and the fact that their prior reasons have been so weak makes me think that if this, of all things, is what they're trying to be vague about, then I assume this must be so unique and specific that even they recognize it's a weak critique.

Re: Scaling Rust Builds with Bazel

#118

Post by dilyevsky »

reitanuki wrote: post #49
bjackman wrote: post #22

Earlier quoted context omitted.

Lol, Googler here, I can't deny that I have he urge to force Bazel into the world. It's just, when I joined Google I found a build system I didn't hate. This was the first time. I don't actually love Bazel (Blaze internally), but I love that I don't hate it. Every time I have to learn a new build system in open source work I groan. Why are there so many systems for essentially the same thing?

on the other hand, there are some of us that really dislike Bazel. Try compiling something for ARM and then realising you can't find a build of the build tool for ARM... That was a special level of pain. Tools commonly available in distros are typically much easier to use for outsiders.

I’m cross-compiling c++/rust/go project to linux arm target from mac hosts. Took 2hrs to setup.

> Tools commonly available in distros are typically much easier to use for outsiders.

Which part about autotools or ninja is easier?

Re: Scaling Rust Builds with Bazel

#119

Post by lern_too_spel »

di4na wrote: post #109
fsociety wrote: post #73

Earlier quoted context omitted.

The standard suite of open sourced Bazel rules are powerful enough for 99% of use cases. I have seen people argue this isn’t true, or that a rules package is fundamentally broken, but then learn that they are holding it wrong. And that holding it wrong reveals some glaring issues with how they setup their build system. It’s not solely the user’s responsibility to hold it right. But Bazel is and has been developed in…

If people hold it wrong so much, the tool is not adapted to the users. By definition

These people hold other build tools wrong too. The difference is those other build tools often have no way of being held correctly. That Bazel is possible to use correctly at all explains why it was copied at least twice (Buck and Pants) before it was open sourced.

Re: Scaling Rust Builds with Bazel

#120

Post by sam_bristow »

cybrexalpha wrote: post #67

I've never seen a project that uses Bazel that didn't also have a backlog task to remove it.

The [Magma project](https://github.com/magma/magma) uses Bazel. The transition seems to have gone well enough they're looking at ripping out the legacy Makefile based build system they maintained in parallel.
Post reply on HN