Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bazel – Correct, reproducible, fast builds for everyone

bazel.io

Post reply on HN
Print view View original

41–50 of 185 posts

Re: Bazel – Correct, reproducible, fast builds for everyone

#41

Post by pron »

hanwenn wrote: post #16
pron wrote: post #13

Earlier quoted context omitted.

Do you also use timestamps like sjavac or some other mechanism, like hashing?

Bazel uses checksums to determine if builds are up-to-date, but shortcuts the checksumming if metadata (timestamp, filesize) have not changed between builds.

Your FAQ says this:

Bazel configuration files are much more structured than Gradle's, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility.

Could you please elaborate on that (i.e. with regards to both parallelism and reproducibility)?

Re: Bazel – Correct, reproducible, fast builds for everyone

#42

Post by thechao »

I've been burned by so many build tools over the years. I've finally settled (for C/++/asm) on the combination of Make + ccache: I build a _very_ paranoid Makefile that recompiles everything if it feels like anything changes. For instance, every rule that compiles a C/++ file is invoked if _any_ header/inc/template file changes. I let ccache do the precise timestamp/check-sum based analysis. The result is that (for large builds I just wish that I had a high-performance replacement for linking that was cross-platform (deterministic mode for ar), and for non-C/++ flows. Writing a deterministic ar is about 20 lines of C-code, but then I have to bake that into the tool in awkward ways. For generalized flows, I've looked at fabricate.py as a ccache replacement, but the overhead of spinning up the Python VM always nukes performance.

Re: Bazel – Correct, reproducible, fast builds for everyone

#43

Post by Symmetry »

yarapavan wrote: post #34

Surprisingly, significant parts of the code is not open source. According to this page, http://bazel.io/docs/governance.html , Is Bazel developed fully in the open? Unfortunately not. We have a significant amount of code that is not open source; in terms of rules, only ~10% of the rules are open source at this point. We did an experiment where we marked all changes that crossed the internal and external code bases ov…

Do they mean that 10% of the original Blaze rules are now open source or that 10% of the Bazel rules they've released are open source?

Re: Bazel – Correct, reproducible, fast builds for everyone

#45

Post by spankalee »

yarapavan wrote: post #34

Surprisingly, significant parts of the code is not open source. According to this page, http://bazel.io/docs/governance.html , Is Bazel developed fully in the open? Unfortunately not. We have a significant amount of code that is not open source; in terms of rules, only ~10% of the rules are open source at this point. We did an experiment where we marked all changes that crossed the internal and external code bases ov…

I don't think you're interpreting that section quite right. That section is talking about whether or not Bazel is fully _developed_ in the open, and the answer is "Unfortunately not".

What they mean is that changes to the internal source of Blaze often involve changes to both the open sourced part, which is Bazel, and the closed parts, which are additional rules that are neither open sourced, nor included in Bazel (Blaze has about 5x as many rules as Bazel).

It's best to make atomic changes, so rather than split the changes, review and submit the open source changes externally, and the closed rules changes internally (which would complicate reviews, testing, syncing and rollbacks), then pull in the external changes, they submit these cross-code-base changes internally, then dump the change into the external repo. The next paragraph on that page makes it clear that the code is open, even if not all of the development process is.

To be clear, all of Bazel is open source and the source is available here: https://github.com/google/bazel

Re: Bazel – Correct, reproducible, fast builds for everyone

#47

Post by mashraf »

Is Google departing from just throwing white papers over the wall and let community figure out the implementation details? blaze white paper was dropped a while ago and there are already two clones in Pants and Buck at Twitter and FB. It would be interesting to see how far off clones are from original implementation.

Re: Bazel – Correct, reproducible, fast builds for everyone

#49

Post by lberki »

yarapavan wrote: post #34

Surprisingly, significant parts of the code is not open source. According to this page, http://bazel.io/docs/governance.html , Is Bazel developed fully in the open? Unfortunately not. We have a significant amount of code that is not open source; in terms of rules, only ~10% of the rules are open source at this point. We did an experiment where we marked all changes that crossed the internal and external code bases ov…

Currently about 60% of our code (in terms of lines of Java code, excluding tests) is open sourced. The rest is glue logic to internal Google systems or build rules that we haven't open sourced. Some of these rules, we are planning to open source in the future, and some others are specific to Google, so they don't really make much sense in the open source tree.

Re: Bazel – Correct, reproducible, fast builds for everyone

#50

Post by astral303 »

setheron wrote: post #36

If i'm sticking to primarily Java; is there a benefit to using Bazel as opposed to Maven / Gradle / Sbt ?

At first impression, unless you have a single gigantic source code base, unlikely. From their FAQ:

>> "Gradle: Bazel configuration files are much more structured than Gradle's, letting Bazel understand exactly what each action does. This allows for more parallelism and better reproducibility"

The value of "more parallelism" depends on the complexity of your Java source code base. I can easily imagine why this extra structure can lead to more parallelism.

However, I am not buying "better reproducibility" without justification or explanation. I've had very reproducible Maven builds for years (and I don't see how Gradle would be different). So I would love to know which aspects are improved upon with this structure, if someone could expand or explain.

Finally, I'm very wary of "much more structure". The worst thing about Maven is its extreme insistence on structure and schema and very specific architecture of your build tasks and components. In contrast, with Gradle, you can freely shape your build scripts to reflect the "build architecture" of your source tree in a minimal, maintainable way. Furthermore, when your application's needs change, refactoring your build is far easier in Gradle, thanks to its internal-DSL style (the build script is code).

If the structure isn't "free", you pay for structure with reduced build script development speed. For Google, it's a tradeoff worth having with that massive source tree.

Post reply on HN