Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redo: A recursive, general-purpose build system

redo.readthedocs.io

Post reply on HN
Print view View original

41–50 of 95 posts

Re: Redo: A recursive, general-purpose build system

#41

Post by joshuamorton »

codedokode wrote: post #20
AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

I don't like using tabs in Makefile syntax. Tabs are in many cases (especially on print) indistinguishable from spaces. Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.

> Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.

This requires that either you implement or shell out to a C/C++ parser as part of the dependency resolution step (which has to happen prior to the actual compilation). This will slow down your compilation a lot unless you're able to parse C/C++ as fast as your make/bazel/whatever language.

Re: Redo: A recursive, general-purpose build system

#42

Post by kylecordes »

dataflow wrote: post #38
qbasic_forever wrote: post #11

Earlier quoted context omitted.

Have you given bazel a serious look? It hits quite a few of the points you mention. There's a steep learning curve and you kind of have to go all in with it, but once you're there it's quite nice.

Bazel is alright (read: generally better than the competition) as long as you're not doing something uncommon with your toolchain or hitting one of its many shortcomings, otherwise it can quickly become a nightmare. It doesn't even make it easy to access the darn executable you just compiled (!) which always baffles me—if I compiled a binary I don't want it hidden away in some random folder, I want it right there so…

For the sake of readers:

Bazel does not put outputs in random folders, it puts them in folders that are a deterministic mapping from the build target plus relative path to the thing being built in the source directory. It is well-suited if you have a huge amount of source code with things being output, but one among many speed bumps when getting started.

Re: Redo: A recursive, general-purpose build system

#43

Post by kylecordes »

packetlost wrote: post #19
lolski wrote: post #15

Earlier quoted context omitted.

I second Bazel. People keep on mentioning how steep the learning curve is, but the conceptual model is really simple, elegant, and intuitive. What is steep is the technical know-hows: 1. When things don't work as expected. For example, while it worked flawlessly with languages that it natively support such as Java, that wasn't the case for other languages such as Javascript or Python. 2. When you have to do something…

Bazel is awful unless you're Google and have a team of devs supporting Bazel.

Of course that is an overstatement, but:

I teach Bazel and help companies adopt it. I’ve noticed that a high percentage of those using Bazel have one or more xooglers on their team.

Re: Redo: A recursive, general-purpose build system

#44

Post by joshuamorton »

kylecordes wrote: post #42
dataflow wrote: post #38

Earlier quoted context omitted.

Bazel is alright (read: generally better than the competition) as long as you're not doing something uncommon with your toolchain or hitting one of its many shortcomings, otherwise it can quickly become a nightmare. It doesn't even make it easy to access the darn executable you just compiled (!) which always baffles me—if I compiled a binary I don't want it hidden away in some random folder, I want it right there so…

For the sake of readers: Bazel does not put outputs in random folders, it puts them in folders that are a deterministic mapping from the build target plus relative path to the thing being built in the source directory. It is well-suited if you have a huge amount of source code with things being output, but one among many speed bumps when getting started.

(also depending on what you're doing, `bazel run ` will invoke the binary directly, and `bazel build ` will, if there's an output artifact, usually print its location)

Re: Redo: A recursive, general-purpose build system

#45

Post by tn1 »

AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

Something a little better than make is makepp [1], although it can be more complex. We use it at $work and it works pretty well.

[1] http://makepp.sourceforge.net/

Re: Redo: A recursive, general-purpose build system

#46

Post by redleader55 »

File system time stamps are an unreliable way to decide if a target needs to be rebuilt. It doesn't work with distributed builds, it doesn't allow instant rebuilds to help with bisections, various file systems might not have not store the same time attributes, even on the same OS, etc. In my experience (large, mono-repo builds) hashes are the way to go and offer the most portable way to map an artefact to a tree of dependencies.

Another problem which is not discussed and is a very important feature for a modern build engine is "sandboxing" - forcing a build rule to declare all inputs, all outputs and to prevent any other kind of FS access and network access during the run of that command. BuildXL from Microsoft does that. Without sandboxing it's pretty hard to enforce a sane set of rules for building your large project.

Re: Redo: A recursive, general-purpose build system

#47

Post by KarlKemp »

AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

I’m quite happy with ruby’s rake for my needs. I mostly use it to define pipelines, often involving data downloads and transformations into datasets for ML. It works well when you work with files, preferably containing a single or few entities each. Have been meaning to wrap some tools to make it handle databases into an extension.

Re: Redo: A recursive, general-purpose build system

#48

Post by ghoward »

gavinhoward wrote: post #40

I feel like DJB's ideas behind redo are actually some of his most poorly thought-out ideas. It seems that he thought, 'Make has . To solve , a new build system should have .' In other words, he seems to just used additive solutions instead of subtractive ones. [1] Most build systems seem to be like that, adding features on top of features. The end result is usually complicated and finicky. I think that's why build sy…

Meant to say: To implement dynamic dependencies, you need a suspending scheduler, so obviously, my build system will have a suspending scheduler.

Re: Redo: A recursive, general-purpose build system

#49

Post by dataflow »

kylecordes wrote: post #42
dataflow wrote: post #38

Earlier quoted context omitted.

Bazel is alright (read: generally better than the competition) as long as you're not doing something uncommon with your toolchain or hitting one of its many shortcomings, otherwise it can quickly become a nightmare. It doesn't even make it easy to access the darn executable you just compiled (!) which always baffles me—if I compiled a binary I don't want it hidden away in some random folder, I want it right there so…

For the sake of readers: Bazel does not put outputs in random folders, it puts them in folders that are a deterministic mapping from the build target plus relative path to the thing being built in the source directory. It is well-suited if you have a huge amount of source code with things being output, but one among many speed bumps when getting started.

I meant random in the colloquial sense (same sense as which a SHA256 is "random"), not a random variable.

Re: Redo: A recursive, general-purpose build system

#50

Post by webstrand »

woodruffw wrote: post #18
AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

Not just a functional language -- I found that my ability to write maintainable (and performance) Makefiles drastically improved when I began to think of Make as a Prolog/Datalog variant with particularly heinous syntax and very limited constraint resolution.

I really like the declarative style of make, to the extent that I've been abusing it as an automation tool. Doing things like checking if a server is alive or setting up a serial port. But due to limitations like depending on mtime and issues with special characters in targets, I've switched some of my more intensive Makefiles to prolog using this: https://github.com/webstrand/robo.

When I find the time I hope to write bindings for netlink for prolog, so that I can declaratively create and configure network namespaces. Complete declarative system configuration using prolog would be a dream come true.

Post reply on HN