Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Zb: An Early-Stage Build System

zombiezen.com

Post reply on HN
Print view View original

101–110 of 132 posts

Re: Zb: An Early-Stage Build System

#101

Post by PROMISE_237 »

zombiezen wrote: post #25
laurentlb wrote: post #9

I'd like to know more about the "Support for non-determinism" and how that differs from other build systems. Usually, build systems rerun actions when at least one of the inputs has changed. Are non-deterministic targets rerun all the time? Also, I'm curious to know if you've considered using Starlark or the build file syntax used in multiple other recent build systems (Bazel, Buck, Please, Pants).

(Hi! I recognize your name from Bazel mailing lists but I forget whether we've talked before.) I'm mostly contrasting from Nix, which has difficulty with poisoning cache when faced with non-deterministic build steps when using input-addressing (the default mode). If zb encounters a build target with multiple cached outputs for the same inputs, it rebuilds and then relies on content-addressing to obtain build outputs…

[dead]

Re: Zb: An Early-Stage Build System

#102

Post by imiric »

nurettin wrote: post #95
pdimitar wrote: post #94

Earlier quoted context omitted.

The part you seem to deliberately miss is that what is obvious to people deeply invested in Nix is not obvious to anyone else. I for one can't trace the train of thought that is going from "intermediate build artifact" and somehow arrives at "derivation". I found out just enough about Nix to reject it. My take is still informed, I simply didn't buy its pitch.

I geniunely thought you knew nothing about derivations and were criticizing the blogger for writing the term in their blog, not the term standard to Nix itself. Which is just as weird to me as complaining about std::string, well why call it a string? it is obviously text!

> Which is just as weird to me as complaining about std::string, well why call it a string? it is obviously text!

It's really not, though. String is a common technical term used in programming languages for many decades. If a new language decided to call them "textrons", _that_ would be weird. And this is the exact thing Nix did with "derivations", "flakes", etc. There is no precedent for these terms in other software, so they're unfamiliar even to its core audience.

It would be different if Nix invented an entirely new branch of technology that didn't have any known precedent. But for a reproducible build system that uses a declarative language? C'mon.

Re: Zb: An Early-Stage Build System

#103

Post by bjourne »

I've been using WAF for ages so naturally I wonder how this system compares to WAF? My experience with build systems is that they all get the easy parts rights. You can compile C and C++ code and they successfully scan header files for dependencies. But FEW get the hard parts rights. E.g., compiling LaTeX with multiple figures, custom fonts and classes, and multiple bib files. It requires correctly interfacing with pdfatex which is a complete PITA as it spews intermediate files everywhere and puts constraints on the current directory. Most build tools can't.

What I want in a build tool is universality. Sometimes a whole directory tree is the dependency of a target. Sometimes it's an url and the build tool should correctly download and cache that url. Sometimes the pre-requisite is training an ML model.

Re: Zb: An Early-Stage Build System

#104

Post by bjourne »

o11c wrote: post #24
Iceland_jack wrote: post #5

I appreciate the link to https://dl.acm.org/doi/10.1145/3236774

Definitely interesting, but it's flat-out wrong about the limitations of `make`. In particular, the `release.txt` task is trivial by adding a dummy rule to generate and include dependencies; see https://www.gnu.org/software/make/manual/html_node/Remaking-... (be sure to add empty rules to handle the case of deleted dynamic dependencies). You can use hashes instead of file modification times by adding a different kind…

I guess you aren't keen on Java then? Complex dynamic dependency graphs aren't difficult for humans to handle or many build tools other than make.

Re: Zb: An Early-Stage Build System

#105

Post by ulbu »

imiric wrote: post #85
ulbu wrote: post #81

Earlier quoted context omitted.

what word would you fit to what a nix derivation is?

I'm not sure, I'm not a Nix expert. The comments here also refer to it as both instructions to build something, as well as the intermediate build artifact. This discussion[1] on the NixOS forums explains it as a "blueprint" or "recipe". So there's clearly a lot of confusion about what it is, yet everyone understands "blueprint", "recipe", or even "intermediate build artifact" if you want to be technical. The same is…

yes, i agree, nix should be considered the bible of bad documentations. it’s very bad at spotlighting the essentials and putting the non-essentials aside. it’s especially surprising for derivations, because nix is really, in the end, a frontend for building derivations. everything else converges on it.

and then i go to nix.dev and derivations are presented after fetchers? no surprise it’s so confusing, even though the concept is quite simple.

a derivation is a dict that is composed of (1) a shell script and (2) environment parameters it will have access to. a nix command will read the derivation, create the environment with only these parameters and execute the script. that’s it.

everything else about nix language is about building derivations. like copying files into its store. for example, evaluating “${pkgs.hello}” will be interpolated into a path. so in your derivation, you can define an env variable “hello = ${pkgs.hello}/bin” and it will be available in your script as “$hello” and will have the value of “/nix/store/-hello/bin”. nix will do the fetching and storing for you. so you can have “command $hello” in your script. neat!

play around with evaluating the ‘derivation’ built-in function.

Re: Zb: An Early-Stage Build System

#107

Post by Ericson2314 »

aseipp wrote: post #55
greener_grass wrote: post #29

I'm excited by this! Quick question: if the build graph can be dynamic (I think they call it monadic in the paper), then does it become impossible to reason about the build statically? I think this is why Bazel has a static graph and why it scales so well.

Buck2 can express dynamic dependencies, so it can capture dynamic compilation problems like C++ modules, OCaml/Fortran modules, etc. in "user space" without built-in support like Bazel requires. The secret to why is twofold. One, your internal build graph can be fully dynamic at the implementation level; rather, it's a matter of how much expressivity you expose to the user in letting them leverage and control the dyn…

Austin I think some of these distinctions are not necessary for the theory.

In https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dyn... there is only an action graph but it is dynamic. Dynamic craft would depend on an entire directory, and thus need to be rebuilt a lot. But when individual files are projected out, there is a new opportunity for early cut-off.

Re: Zb: An Early-Stage Build System

#108

Post by JasonSage »

mikepurvis wrote: post #52
Rucadi wrote: post #18

Earlier quoted context omitted.

Why are flakes such a deal-breaker? While not ideal, you can still tag your versions in the .nix file instead of the lockfile. I even had to avoid flakes in a system I developed used by ~200 developers since it involved a non-nixos OS and it involved user secrets (Tokens etc...) So with flakes I had to keep track of the secrets (and was a pain point, since they obviously didn't have to push them into the git repo) bu…

As a new user, I learned flakes first, and the tie-in with git tags/branches and the corresponding cli ergonomics aren’t something I’d be able to give up.

How do you handle flakes pushing an entire copy of the repo into the nix store? Is this not an issue for you somehow?

Re: Zb: An Early-Stage Build System

#109

Post by IshKebab »

israrkhan wrote: post #38
IshKebab wrote: post #17

Interesting. I feel like I would have gone with Starlark over Lua, but I guess it's good to have options. Does it support sandboxing?

You need bazel if you need starlark & sandboxing

Well yeah. Starlark and sandboxing are the best things about Bazel, but it could still definitely be improved. So I'm still curious about other build systems.

I think making a new build system without sandboxing (or at least a plan for it) would be pretty stupid.

Fortunately he is planning it.

Re: Zb: An Early-Stage Build System

#110

Post by rwmj »

bjourne wrote: post #103

I've been using WAF for ages so naturally I wonder how this system compares to WAF? My experience with build systems is that they all get the easy parts rights. You can compile C and C++ code and they successfully scan header files for dependencies. But FEW get the hard parts rights. E.g., compiling LaTeX with multiple figures, custom fonts and classes, and multiple bib files. It requires correctly interfacing with p…

I wrote an experimental make replacement some years ago that understands that not every target is a file. eg. You can have targets be a remote URL (for an action of uploading to a fileserver).

http://git.annexia.org/?p=goals.git;a=summary

http://oirase.annexia.org/2020-02-rjones-goals-tech-talk.mp4

Post reply on HN