Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bazel 2.0

blog.bazel.build

Post reply on HN
Print view View original

61–70 of 117 posts

Re: Bazel 2.0

#61

Post by kylecordes »

shrewduser wrote: post #33
zellyn wrote: post #15

Earlier quoted context omitted.

If you are an organization with a large enough codebase (especially if it's in a monorepo) that you need a shared remote cache of build artifacts, or remote build sharding and execution, and have multiple languages (even protocol buffers) interacting in complex dependencies, then you should run as fast as you can away from less rigorous Blaze-alikes (Pants, Buck, etc.) straight towards Bazel. Yes, it's complicated, b…

I'll never understand the fascination with mono repo's.

At a certain size, monorepo becomes the worst way to do it except for all the others.

Essentially: version skew across numerous artifacts in a large organization starts to look like the version skew across an industry or ecosystem. The aggregate cost of dealing with it project by project is probably higher, at least that is what most of the biggest tech companies have concluded, than dealing with it at the source level using a monorepo and single-version policy.

Re: Bazel 2.0

#62

Post by habitue »

Good heuristic for whether it's worth considering moving to bazel for your build system:

- Do you have 200+ developers working on a monorepo?

- Are you willing to vendor all of your dependencies and maintain their builds yourself?

If so, consider it. The productivity you're losing to unnecessary rebuilding and re-running unchanged unit tests will probably be paid back if you can contort your development process to the one Bazel expects.

If you're a small shop, the benefits Bazel is going to provide over, say, Make (or whatever standard build system your primary language uses), are going to be minimal. And the overhead of maintaining Bazel is going to cost you a ton of developer time you may not be able to afford.

Re: Bazel 2.0

#63

Post by djsumdog »

I hadn't heard of this, and see there is a lot of concern over using this project except for specific use cases.

I'm always weary of build tools that try to do multiple languages. On Scala projects I use SBT, and for people who have tried to hack on SBT itself or its plugins, you know it's a big mess under there. On other projects I've tried using Gradle with Scala, but I found a lot of times Gradle just wasn't setup for a Scala workflow or was missing essential tooling to make it as effective as SBT (although its configuration is considerably more sane). Most of the tooling and plugins around Scala are built around SBT as well (for better or for worse).

I try to stick with the major tool for a given language; cargo for Rust, SBT for Scala, the built-in tooling for go, with the exception of Java projects where I'd gladly take gradle over the hellscape that is Maven.

Re: Bazel 2.0

#64

Post by habitue »

zellyn wrote: post #56
arianvanp wrote: post #47

Earlier quoted context omitted.

Nix is not (yet) suitable for fine-grained (read file level) build targets though due to lack of recursive nix and content addressed store. This means you don't have early cut off and mass rebuilds if only one file changes; for example. Both are being worked on actively though.

If I understand it correctly (unlikely), Nix has the degree of purely-functional rigor necessary to do this correctly, right? Sounds like it would eventually be awesome for Bazel usecases.

Nix isn't great as a build system, because it throws a lot out and rebuilds everything when something changes. It's intended to get a correct, isolated package installed, not to maximize sharing.

Bazel goes through great pains to only rebuild the minimum necessary for correctness. It's able to do that because bazel build files get a lot more information about the source level dependencies than a Nix file does

Re: Bazel 2.0

#65

Post by [deleted] »

lenkite wrote: post #52
teeray wrote: post #34

Earlier quoted context omitted.

I miss the days when JavaScript frameworks required only a script tag.

Those days are BAAACK. Use Vue ! (The best JS framework in the world!) import Vue from ' https://unpkg.com/vue@2.6.0/dist/vue.esm.browser.min.js'; new Vue({ ... });

[deleted]

Re: Bazel 2.0

#66

Post by meddlepal »

downerending wrote: post #37
simfoo wrote: post #18

Earlier quoted context omitted.

Pretty much the same experience. 2y old monorepo company with >150 engineers now, being slowed down by hacks upon hacks in the usage of the bazel build system with around 5 people at the company understanding how bazel works. And those people are discouraged quickly when trying to improve things because of the shear brittleness of the existing build files and "optionated" approaches in the community that don't quite…

Indeed. If Bazel had a slogan, it would be "Everyone who doesn't do things my way is stupid". That's probably great within Google, but trying to integrate it into a company that has already made conflicting decisions (for good reasons) is hell.

I feel like that slogan could apply to most OSS projects Google dumps on the world.

Re: Bazel 2.0

#67

Post by breatheoften »

Does bazel use the word “provenance“ at all?

Provenance is a word I first saw advertised in a platform called dotscience.io — that I find fundamentally interesting. And it seems quite relevant to hermetic builds.

Provenance is about giving any state derived from an arbitrary computation an identity that is derived from the content hash of the inputs needed to re-compute that state ... in dotscience they achieve this by instrumenting io and creating zfs filesystem snapshots when computing new provenance artifacts.

I think this concept could be the ultimate building block for a build system — and it could become the job of oses/containers/runtimes/databases to Coordinate to allow this abstraction to be tracked with sufficient efficiency that programmers would feel allowed to freely use the concept of provenance when building ... it seems to me like provenance could provide all the information needed to support a distributed build cache? You wouldn’t actually need a build language at all — just an api in each language to ask for the saving of provenance artifacts. The artifact would hold all the info needed to be able to recompute the artifact with the same state — which is also all the info needed to decide when the artifact is out of date ...?

Re: Bazel 2.0

#68

Post by vlovich123 »

boris wrote: post #57
ajconway wrote: post #39

Earlier quoted context omitted.

> run as fast as you can in the opposite direction But which one? Are there any other (non blaze-like) build systems enabling hermetic (possibly remote) builds and caching? If you don't need these properties and have a mono language project, the language's native build system sure fits and may be a better choice.

There is build2. It has "high-fidelity" (instead of hermetic) builds meaning that besides sources it keeps track of changes to options, compilers, etc. This gives you similar benefits at a fraction of the cost. There is no distributed compilation or caching yet but it's coming. In other benefits, it doesn't need Java or Python (or any other "platform").

Assuming you didn't actually care about hermetic builds, the challenge with build2 is that it's c++ only AFAIK. Larger orgs turn into polyglot scenarios (python/bash for scripting/gluing things together, Go for web services, C/C++ for high performance code, now Rust, etc).

You can of course try to use the native solution on each but that makes it more difficult for people to jump between projects/languages as the syntax for describing the build changes. Moreover for centralized build infra this becomes more difficult to orchestrate/co-ordinate because now you have to add remote caching/parallel compilation & whatnot to multiple places (with all the associated challenges of trying to upstream the same set of logical changes into many different projects with their own maintenance schedules/philosophies).

Re: Bazel 2.0

#69

Post by bobdobbs666 »

I was subjected to bazel on a small project because the manager insisted we use it. The rest of the company used a number of either custom tools or cmake or premske.

It is utter hell when you have tons of third party libraries (internal or external to the company) that you don’t have the source to and it is especially painful when trying to integrate bazels behavior against other build systems. Also bazels packaging and use of internal symlink renaming was a constant source of suffering. Bazel pretty much destroys a number of totally valid work linux commands for looking for so files.

Bazel might be useful in the case of a monorepo with a massive engineering pool AND a massive cloud infrastructure backing that repo to handle all the artifact sharing, but after having used cmake, premake, waf, random perl and ruby scripts, or just checking vs projects into perforce manually, I’d pick any of those before bazel for most projects. I say that having worked on code bases from a few 10s of thousands to 25+ million LoC with teams small, large, and distributed.

Bazel probably has its place but I have yet to find it.

Re: Bazel 2.0

#70

Post by malkia »

One issue we hit with our CI, and mix of build systems is this - given a changelist, find out which targets needs to be built, and which one needs to be tested on pre-submit, and which on post-submit.

With that, we end up paying so much extra time building everything over and over without need, and then not building things that we ought to.

So that's one reason to switch, but at the same time lots of people simply do not get it. To them it seems intrusive, new, opinionated, and makes them not happy to use it. I've used it for 2+ years at google, and yes initially - was WTF is this? Then it hit me... And I'm sure the same is for buck, pants, please.build, gn and other similar systems.

At the end of the day, you need way to express "end to end" your build graph, from any single individual source file, shell script, or configuration downto building your executables, deploying them, etc.

It's an industry tool, that needs to be looked, and if it takes 5 people to support it, then it takes 5 people to support it, but you won't be wasting other peeople's time on issues like - "Why this build in the CI did not trigger?", why it takes, and wastes my time (waiting for presubmit), etc.

Yes, it does not come for free, but it's worth knowing and trying it out at least.

If nothing else, here is the takeaway - Try to use a system with static graph, where relationships are known before you start building things. It's not always there, e.g. your #include "header.h" file is dynamic, but bazel forces you to express even that, and later it catches it whether you've done it, and breaks unless it's fixed.

Post reply on HN