Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bazel Fawlty

danmux.com

Post reply on HN
Print view View original

11–20 of 52 posts

Re: Bazel Fawlty

#11

Post by mnw21cam »

philbarr wrote: post #7
sarabande wrote: post #3

Thanks for the comments on Bazel -- I am evaluating a proper build system for us right now for a data science codebase almost entirely in Python, with a lot of scientific computing dependencies. Bazel doesn't yet seem to have first-class support for Python, so that's out, and I'm running into a lot of problems with Twitter Pants, although their Slack channel is really helpful. Did you ever try using Pants or Buck, or…

Classic Basil Fawlty scene when his car breaks down (needs sound, includes swearing): https://www.youtube.com/watch?v=mv0onXhyLlE Imagine the car is your build tool...

Which more recently gave rise to this [0] advert.

[0] https://www.youtube.com/watch?v=n9tSN0178Us

Re: Bazel Fawlty

#12

Post by danmux »

sarabande wrote: post #3

Thanks for the comments on Bazel -- I am evaluating a proper build system for us right now for a data science codebase almost entirely in Python, with a lot of scientific computing dependencies. Bazel doesn't yet seem to have first-class support for Python, so that's out, and I'm running into a lot of problems with Twitter Pants, although their Slack channel is really helpful. Did you ever try using Pants or Buck, or…

Glad it can help show you some of the pitfalls, though I'm not sure how well our experience extrapolates to a primarily python codebase.

I have added a link to the classic scene as referenced by philbarr

Re: Bazel Fawlty

#13

Post by danmux »

pvg wrote: post #4

What's the context for this - as in, the 'we' in question and the thing they were building with bazel? And are they really posting from March 16, 2018?

I probably should have made the context clearer earlier in the post - later on I mention :

"To set the scene, our codebase is mainly Go, including vendor directories, and a considerable amount of data compiled in, we have 2.5M lines of code, a full build from a clean clone on one of our Jenkins slaves takes 1 minute.

We had between 8 and 20 engineers working on that codebase. Importantly we all develop on Mac, but run in production in linux."

Re: Bazel Fawlty

#14

Post by leg100 »

erain wrote: post #6

For your docker container needs: "Bazel should in theory be able to decide if a docker image should be rebuilt" You can checkout Bazel's docker rule: https://github.com/bazelbuild/rules_docker Also, go already has a very good build sysmtem build-in, and hazel really shines when: - you have a complex codebase with multi-languages: one can build a tool in one language and use it as tool for a another language. - you si…

It eschews docker, and Dockerfiles and their non-determinism, and builds the layers itself, and uses a python lib [1] to push and pull container images.

In order to ensure reproducibility/determinism, however, Bazel doesn't have an equivalent of the RUN instruction. You have to use other Bazel rules to fetch and produce artefacts and add them to an image, and there aren't always rules for what it is you want to do (I have a spot of bother with installing pip packages, for which there is apparently alpha support).

This is, I think, the thing with Bazel: it has to re-invent everything to ensure this reproducibility and hermetic seal, because it doesn't trust existing build tools to do this, and quite rightly so if this is what you're seeking. But I suspect it's going to be painful doing things outside the mainstream supported stuff.

[1] https://github.com/google/containerregistry

Re: Bazel Fawlty

#15

Post by danmux »

sarabande wrote: post #3

Thanks for the comments on Bazel -- I am evaluating a proper build system for us right now for a data science codebase almost entirely in Python, with a lot of scientific computing dependencies. Bazel doesn't yet seem to have first-class support for Python, so that's out, and I'm running into a lot of problems with Twitter Pants, although their Slack channel is really helpful. Did you ever try using Pants or Buck, or…

Also as you work with Python, you may want to look at the link between Basil Fawlty and your chosen language :)

Re: Bazel Fawlty

#16

Post by danmux »

venantius wrote: post #8

I have worked extensively with Bazel in the context of migrating a very large Java/Scala codebase from Gradle to Bazel. My impression is that it is a first-class build system _specifically for Java and C++_. There are specific properties of the compilation and packaging ecosystem around those languages and runtimes that make them uniquely in need of tools like Bazel. This is not true for Go, where build speed and lar…

Spot on.

Re: Bazel Fawlty

#17

Post by Sir_Cmpwn »

We've been using Bazel at work, where we have a monorepo with lots of different languages at play (Python, Go, Java, Scala, JavaScript, CoffeeScript, TypeScript, Protobuf, just to name the major players) and I like that it makes all of them use a consistent interface for configuring and running builds. It helps that we have a specific team handling most of the Bazel work - and it costs about 10% of all of that team's time.

That's about the only thing I like about it, though. It's the most complicated build system I've ever used and I still don't fully grok it after 7 months working in this position.

Re: Bazel Fawlty

#18

Post by elteto »

It seems like Bazel's support for Go is still not first class and a lot of the issues mentioned here come from that.

But it strikes me as odd that they are figuring all of this out _now_, as opposed to a year ago when they first switched. Why did they switch to bazel in the first place? Did they do a trade study/test drive?

All of the problems mentioned (more complex build files and workflow, increased build times, lacking support for Go) you can find out right away as soon as you do a test drive. Unlike C++ or Java, it seems like the native go build tooling was always superior to what Bazel could offer, and they knew that. Why switch then?

In my experience using Bazel in a large and complex C++ codebase, it has been nothing short of amazing, compared to the CMake horror show that we had... We did have problems with it though, but mostly the lacking documentation in some areas, like building toolchain files, or the fact that support for Windows only got really good a couple of releases ago only. We did identify all of these in the initial migration effort and none of them was by itself a showstopper.

Re: Bazel Fawlty

#19

Post by philip1209 »

When we used Bazel at Staffjoy, we had two main issues:

(1) NPM dependency management was hard. We ended up committing built dist files, like index.html

(2) We ended up duplicating dependencies between Bazel and Glide because tools like goimports and linters could not read the Bazel remote dependencies system.

We open-sourced the repo with Bazel when we shut down -> http://github.com/staffjoy/v2

Re: Bazel Fawlty

#20

Post by zellyn »

If your entire codebase is Go, you really shouldn't be using Bazel/Pants/Buck/Please. Go's built-in build system is perfectly adequate for just Go, with light tooling for generated code (eg. protobufs).

It's when you have multiple languages and want to be able to build/test only what has changed that you need a Blaze-alike.

With regards to Gazelle: is it intended to be run every build, on every file? Or is it intended to be hooked into your editor, and run on save to update your BUILD files as you go? At Google, I used the latter kind of tool, and it worked just fine. If it missed something, you'd notice when the CI system failed to build/test your change.

Post reply on HN