Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zb: An Early-Stage Build System

zombiezen.com

Post reply on HN
Print view View original

21–30 of 132 posts

Re: Zb: An Early-Stage Build System

#21

Post by droelf »

Rucadi wrote: post #16
msvan wrote: post #10

As a current Nix user, what I would really like is a statically typed language to define builds. Recreating Nix without addressing that feels like a missed opportunity.

For me the killer feature is Windows Support, Ericsson is doing a great job bringing nix into Windows, but the process it's understandably slow, If this project is similar enough to nix that I can kind-off translate easily the zb derivations to nix derivations, I'm willing to use it in windows (It's not like nix has windows programs in the nixpkgs either way I have to bring them in my own). The problem for me is that…

We're working on rattler-build (https://github.com/prefix-dev/rattler-build/) - which is a build system inspired by Apko / conda-build and uses YAML files to statically define dependencies. It works really well with pixi (our package manager) but also any other conda compatible package managers (mamba, conda).

And it has Windows support, of course. It can also be used to build your own distribution (e.g. here is one for a bunch of Rust utilities: https://github.com/wolfv/rust-forge)

Re: Zb: An Early-Stage Build System

#23

Post by droelf »

Cool that this space is getting more attention - I just came from the reproducible builds summit in Hamburg. We're working on similar low level build system tools with rattler-build and pixi. Would love to have a chat and potentially figure out if collaboration is possible.

Re: Zb: An Early-Stage Build System

#24

Post by o11c »

Iceland_jack wrote: post #5

I appreciate the link to https://dl.acm.org/doi/10.1145/3236774

Definitely interesting, but it's flat-out wrong about the limitations of `make`.

In particular, the `release.txt` task is trivial by adding a dummy rule to generate and include dependencies; see https://www.gnu.org/software/make/manual/html_node/Remaking-... (be sure to add empty rules to handle the case of deleted dynamic dependencies). You can use hashes instead of file modification times by adding a different kind of dummy rule. The only downside is that you have to think about the performance a little.

I imagine it's possible for a project to have some kind of dynamic dependencies that GNU make can't handle, but I dare say that any such dependency tree is hard to understand for humans too, and thus should be avoided regardless. By contrast, in many other build tools it is impossible to handle some of the things that are trivial in `make`.

(if you're not using GNU make, you are the problem; do not blame `make`)

Re: Zb: An Early-Stage Build System

#25

Post by zombiezen »

laurentlb wrote: post #9

I'd like to know more about the "Support for non-determinism" and how that differs from other build systems. Usually, build systems rerun actions when at least one of the inputs has changed. Are non-deterministic targets rerun all the time? Also, I'm curious to know if you've considered using Starlark or the build file syntax used in multiple other recent build systems (Bazel, Buck, Please, Pants).

(Hi! I recognize your name from Bazel mailing lists but I forget whether we've talked before.)

I'm mostly contrasting from Nix, which has difficulty with poisoning cache when faced with non-deterministic build steps when using input-addressing (the default mode). If zb encounters a build target with multiple cached outputs for the same inputs, it rebuilds and then relies on content-addressing to obtain build outputs for subsequent steps if possible. (I have an open issue for marking a target as intentionally non-deterministic and always triggering this re-run behavior: https://github.com/256lights/zb/issues/33)

I'll admit I haven't done my research into how Bazel handles non-determinism, especially nowadays, so I can't remark there. I know from my Google days that even writing genrules you had to be careful about introducing non-determinism, but I forget how that failure mode plays out. If you have a good link (or don't mind giving a quick summary), I'd love to read up.

I have considered Starlark, and still might end up using it. The critical feature I wanted to bolt in from Nix was having strings carrying dependency information (see https://github.com/NixOS/nix/blob/2f678331d59451dd6f1d9512cb... for a description of the feature). In my prototyping, this was pretty simple to bolt on to Lua, but I'm not sure how disruptive that would be to Starlark. Nix configurations tend to be a bit more complex than Bazel ones, so having a more full-featured language felt more appropriate. Still exploring the design space!

Re: Zb: An Early-Stage Build System

#26

Post by zombiezen »

alxmng wrote: post #22

Did you consider writing a nicer language that compiles to Nix? A "friendly" tool on the outside with Nix inside.

Yup, that was how I built the prototype: https://www.zombiezen.com/blog/2024/06/zb-build-system-proto...

The last commit using that approach was https://github.com/256lights/zb/tree/558c6f52b7ef915428c9af9... if you want to try it out. And actually, I haven't touched the Lua frontend much since I swapped out the backend: the .drv files it writes are the same.

The motivation behind replacing the backend was content-addressibility and Windows support, which have been slow to be adopted in Nix core.

Re: Zb: An Early-Stage Build System

#27

Post by Rucadi »

alxmng wrote: post #22

Did you consider writing a nicer language that compiles to Nix? A "friendly" tool on the outside with Nix inside.

I don't think nix is that awful, while there are some tasks that are more difficult or can be a little bit verbose (if you want to play a lot with the attribute sets / lists or string manip) When using nix most of the time you'll end up just writing bash or using it as a templating language.

Re: Zb: An Early-Stage Build System

#28

Post by zombiezen »

mikepurvis wrote: post #12

Whoa, nifty. Can you speak more to the interop issues with Nix? I've been working on a pretty large Nix deployment in the robotics space for the past 3ish years, and the infrastructure side is the biggest pain point: * Running a bare `nix build` in your CI isn't really enough— no hosted logs, lack of proper prioritization, may end up double-building things. * Running your own instance of Hydra is a gigantic pain; it'…

Tracking issue is https://github.com/256lights/zb/issues/2

The hurdles to interop I see are:

- Nixpkgs is not content-addressed (yet). I made a conscious decision to only support content-addressed derivations in zb to simplify the build model and provide easier-to-understand guarantees to users. As a result, the store paths are different (/zb/store instead of /nix/store). Which leads to... - Nix store objects have no notion of cross-store references. I am not sure how many assumptions are made on this in the codebases, but it seems gnarly in general. (e.g. how would GC work, how do you download the closure of a cross-store object, etc.) - In order to obtain Nixpkgs derivations, you need to run a Nix evaluator, which means you still need Nix installed. I'm not sure of a way around this, and seems like it would be a hassle for users.

I have experienced the same friction in build infra for Nix. My hope is that by reusing the binary cache layer and introducing a JSON-RPC-based public API (already checked in, but needs to be documented and cleaned up) for running builds that the infrastructure ecosystem will be easier.

Re: Zb: An Early-Stage Build System

#29

Post by greener_grass »

I'm excited by this!

Quick question: if the build graph can be dynamic (I think they call it monadic in the paper), then does it become impossible to reason about the build statically? I think this is why Bazel has a static graph and why it scales so well.

Re: Zb: An Early-Stage Build System

#30

Post by zombiezen »

steeleduncan wrote: post #8

Looks great, Nix-with-Lua that also supports Windows would be amazing. Two questions if I may - Does this sandbox builds the way flakes do? - What is MinGW used for on Windows? Does this rely on the MinGW userland, or is it just because it would be painful to write a full bootstrap for a windows compiler while also developing Zb? Also, its great to see the live-bootstrap in there. I love the purity of how Guix's pack…

Nix sandboxes derivation runs on Linux even without flakes, and I'm planning on implementing that, yes: https://github.com/256lights/zb/issues/29 and https://github.com/256lights/zb/issues/31

MinGW is used to build Lua using cgo. I'd like to remove that part, see https://github.com/256lights/zb/issues/28 I haven't started the userspace for Windows yet (https://github.com/256lights/zb/issues/6), but I suspect that it will be more "download the Visual C++ compiler binary from this URL" than the Linux source bootstrap.

Yeah, I'm happy with live-bootstrap, too! I tried emulating Guix's bootstrap, but it depended a little too much on Scheme for me to use as-is. live-bootstrap has mostly worked out-of-the-box, which was a great validation test for this approach.

Post reply on HN