Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bazel Fawlty

danmux.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Bazel Fawlty

#31

Post by jasikpark »

sarabande wrote: post #3

Thanks for the comments on Bazel -- I am evaluating a proper build system for us right now for a data science codebase almost entirely in Python, with a lot of scientific computing dependencies. Bazel doesn't yet seem to have first-class support for Python, so that's out, and I'm running into a lot of problems with Twitter Pants, although their Slack channel is really helpful. Did you ever try using Pants or Buck, or…

Using `pipenv`[0] may solve some of your problems, though idk about provisioning cpp dependencies. [0]https://github.com/pypa/pipenv

Re: Bazel Fawlty

#32

Post by cromwellian »

venantius wrote: post #8

I have worked extensively with Bazel in the context of migrating a very large Java/Scala codebase from Gradle to Bazel. My impression is that it is a first-class build system _specifically for Java and C++_. There are specific properties of the compilation and packaging ecosystem around those languages and runtimes that make them uniquely in need of tools like Bazel. This is not true for Go, where build speed and lar…

Actually it is very relevant for JavaScript at Google, given Closure Compiler. Most of our JS apps are type checked, code split and optimized and Bazel is how we structure the module dependencies. The Bazel build rules define independent logical modules, and build processes prune unused dependencies, and calculate a dependency graph which is used to organize an optimal module structure, this is then used by Closure Compiler’s CrossModuleCodeMotion to further reduce size by moving methods and properties down to later modules.

Granted, JS isn’t incrementally built, but the dependency graph pruning (removing unused files from the build that aren’t imported or referenced in JS code) is parallelizable.

Re: Bazel Fawlty

#33

Post by danmux »

zellyn wrote: post #29
danmux wrote: post #26

Earlier quoted context omitted.

Regarding Gazelle, yes, the CI would break early if it detected a none porcelain repo after running Gazelle itself. I think others may have configured their editors, but for me the occasional 30+s run time (and as mentioned in some workflows 3-4 mins) on file save - plus the already creaking VSCode plugins was too much to bear.

Was it 3-4 minutes when updating a single file? Or only running over the whole repo?

yes, could easily have been the latter

Re: Bazel Fawlty

#35

Post by cyphar »

leg100 wrote: post #14
erain wrote: post #6

For your docker container needs: "Bazel should in theory be able to decide if a docker image should be rebuilt" You can checkout Bazel's docker rule: https://github.com/bazelbuild/rules_docker Also, go already has a very good build sysmtem build-in, and hazel really shines when: - you have a complex codebase with multi-languages: one can build a tool in one language and use it as tool for a another language. - you si…

It eschews docker, and Dockerfiles and their non-determinism, and builds the layers itself, and uses a python lib [1] to push and pull container images. In order to ensure reproducibility/determinism, however, Bazel doesn't have an equivalent of the RUN instruction. You have to use other Bazel rules to fetch and produce artefacts and add them to an image, and there aren't always rules for what it is you want to do (I…

> It eschews docker, and Dockerfiles and their non-determinism, and builds the layers itself [...] But I suspect it's going to be painful doing things outside the mainstream supported stuff.

This should not matter at all. The OCI specification provides an interoperable image format that doesn't care who built the image or how. Docker still doesn't support the OCI yet[1], but tools exist to do conversions[2]. To shamelessly plug my own projects, you can use umoci[3] as a completely unprivileged user to construct images (without using containers if you don't want to) and the resulting images are still usable with Docker (once you convert them from OCI images). This is how we build the official openSUSE images, and it doesn't affect any user of the images that we chose not to use Docker or Dockerfiles to build them.

[1]: https://github.com/moby/moby/pull/33355 [2]: https://github.com/projectatomic/skopeo [3]: https://github.com/openSUSE/umoci

Re: Bazel Fawlty

#36

Post by Aissen »

mkobit wrote: post #27
Aissen wrote: post #22

It added a dependency on the JVM. (I probably have an overly painful past with the JVM in production), suffice to say we had to bump up the RAM on the build machines. This alone is a reason not to use Bazel. It's designed for the enterprise in mind, where JVMs are common, sharing a common build environment in a team is easy (or at least, the setup time isn't an issue). For a community project, this type of thing woul…

> There's a reason Meson is winning the latest build-systems war: What do you mean by winning? I haven't heard of Meson until the last few weeks, and I haven't seen it consumed by any open source projects on GitHub, although my search biases more towards JVM things anyways. > it only depends on python3 & ninja I don't see how depending on both Python3 and Ninja is any less dependencies than the JVM. Python3 isn't rea…

See https://youtu.be/gHdTzdXkhRY?t=8m4s for a list of high-profile projects that have moved or are moving to meson.

While my parent comment is over-simplifying, the talk goes into more details on the strengths of Meson.

Also, I'm not saying there's no open-source community around JVM-based projects. Just that adding it as a dependency is a very expensive decision to make. Python3 is pre-installed in major distros (e.g in Fedora/Ubuntu/Arch), and the binary package for ninja is about 300k installed here. Python3 is about 10m; openjdk is about 100M. Then the runtime requirements the article talks about add up.

Re: Bazel Fawlty

#37

Post by leg100 »

cyphar wrote: post #35
leg100 wrote: post #14

Earlier quoted context omitted.

It eschews docker, and Dockerfiles and their non-determinism, and builds the layers itself, and uses a python lib [1] to push and pull container images. In order to ensure reproducibility/determinism, however, Bazel doesn't have an equivalent of the RUN instruction. You have to use other Bazel rules to fetch and produce artefacts and add them to an image, and there aren't always rules for what it is you want to do (I…

> It eschews docker, and Dockerfiles and their non-determinism, and builds the layers itself [...] But I suspect it's going to be painful doing things outside the mainstream supported stuff. This should not matter at all. The OCI specification provides an interoperable image format that doesn't care who built the image or how. Docker still doesn't support the OCI yet[1], but tools exist to do conversions[2]. To shame…

> This should not matter at all. The OCI specification provides an interoperable image format that doesn't care who built the image or how

The OCI doesn't, but the build tool does care about how the image was built, because of reproducibility etc. But build tools will be able to leverage the kind of stuff you're doing, rather than doing it themselves (like Bazel does).

- umoci looks pretty cool, this is how things should have been done in the first place.

Re: Bazel Fawlty

#38

Post by venantius »

cromwellian wrote: post #32
venantius wrote: post #8

I have worked extensively with Bazel in the context of migrating a very large Java/Scala codebase from Gradle to Bazel. My impression is that it is a first-class build system _specifically for Java and C++_. There are specific properties of the compilation and packaging ecosystem around those languages and runtimes that make them uniquely in need of tools like Bazel. This is not true for Go, where build speed and lar…

Actually it is very relevant for JavaScript at Google, given Closure Compiler. Most of our JS apps are type checked, code split and optimized and Bazel is how we structure the module dependencies. The Bazel build rules define independent logical modules, and build processes prune unused dependencies, and calculate a dependency graph which is used to organize an optimal module structure, this is then used by Closure C…

Yeah, JavaScript is a bit of an odd one out here, and I can obviously see the value in terms of providing a minimal build (in a way that's of less import to Ruby/Python users, say).

That said, it all depends what benefits you want to get out of Bazel. For us, incremental builds were the key thing we cared about - minimizing JAR size was important, but we already had a decently good handle on that.

Re: Bazel Fawlty

#39

Post by davidstanke »

sarabande wrote: post #3

Thanks for the comments on Bazel -- I am evaluating a proper build system for us right now for a data science codebase almost entirely in Python, with a lot of scientific computing dependencies. Bazel doesn't yet seem to have first-class support for Python, so that's out, and I'm running into a lot of problems with Twitter Pants, although their Slack channel is really helpful. Did you ever try using Pants or Buck, or…

Bazel + Python definitely needs some work; if you're interested, check out the recently-formed https://groups.google.com/forum/#!forum/bazel-sig-python to track progress.

Re: Bazel Fawlty

#40

Post by mkobit »

Aissen wrote: post #36
mkobit wrote: post #27

Earlier quoted context omitted.

> There's a reason Meson is winning the latest build-systems war: What do you mean by winning? I haven't heard of Meson until the last few weeks, and I haven't seen it consumed by any open source projects on GitHub, although my search biases more towards JVM things anyways. > it only depends on python3 & ninja I don't see how depending on both Python3 and Ninja is any less dependencies than the JVM. Python3 isn't rea…

See https://youtu.be/gHdTzdXkhRY?t=8m4s for a list of high-profile projects that have moved or are moving to meson. While my parent comment is over-simplifying, the talk goes into more details on the strengths of Meson. Also, I'm not saying there's no open-source community around JVM-based projects. Just that adding it as a dependency is a very expensive decision to make. Python3 is pre-installed in major distros (e.…

I wasn't aware that Python3 was pre-installed on major distros, I don't believe it was by default on my Ubuntu 16.04. It is possibly I removed it, though. It does look like 18.04 is going to bundle Python 3 [1]. I agree with you that adding JVM is an expensive decision and the runtime requirements can make using them in real projects a big pain.

Thanks for the video, I'll check it out. It is exciting (and I would also say a bit worrying) to see a lot of competing tools in this area.

[1] https://wiki.ubuntu.com/Python/Python36Transition

Post reply on HN