Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redo: A recursive, general-purpose build system

redo.readthedocs.io

Post reply on HN
Print view View original

61–70 of 95 posts

Re: Redo: A recursive, general-purpose build system

#61

Post by glandium »

JdeBP wrote: post #56
sigjuice wrote: post #32

Earlier quoted context omitted.

make doesn't really understand C or C dependencies, but the compilers usually do. Here is a minimal example that will take care of dependencies by passing -MD -MP to gcc or clang. The dependencies (*.d files) get generated as part of the normal gcc/clang invocations. There is no separate dependency generation step. $ cat Makefile CFLAGS := -MD -MP hello: hello.o x.o y.o -include *.d $ echo '#include "x.h"' > x.c $ ec…

Actually, the compilers do not and you are doing less than half of the job there. They don't emit information about the non-existent files that would change the build if they were to suddenly exist, such as an "x.h" in one of several standard search directories in that example. * http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDefic...

Also they don't add an empty rule for the source file (like they can do for headers), so if you rename from e.g. foo.c to foo.cpp, which still produces a foo.o, your incremental build fails because the dep file still has the dep on foo.c which doesn't exist anymore.

Re: Redo: A recursive, general-purpose build system

#62

Post by dotancohen »

kylecordes wrote: post #43
packetlost wrote: post #19

Earlier quoted context omitted.

Bazel is awful unless you're Google and have a team of devs supporting Bazel.

Of course that is an overstatement, but: I teach Bazel and help companies adopt it. I’ve noticed that a high percentage of those using Bazel have one or more xooglers on their team.

  > I teach Bazel and help companies adopt it.
The very fact that this occupation exists pushes me away from Bazel, having never used it. Makefiles are horrible, but they are well documented and I've been able to google my way out of every problem that I've either run into, inadvertently caused, or had to debug from somebody else's mess (it's always a mess).

I don't actually want a better build tool. I want a tool that I spend as little time as possible thinking about.

Re: Redo: A recursive, general-purpose build system

#63

Post by 1vuio0pswjnm7 »

AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

"As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement."

Unless I am mistaken, the author of https://cr.yp.to/redo.html never implemented the idea. He has since used only the Bourne shell and standard UNIX utilties as a build system instead of make. For example, consider the "do" script in this project: https://nacl.cr.yp.to

The "do" script was written circa 2011. The idea for "redo" was published circa 2003. The idea for "make" dates back to 1976. GNU Make was written around 1989. For the avoidance of doubt, I am not suggesting any of these build system ideas or implementations are better or worse than the others. That is for the reader to decide. I am only pointing out which one is the most recent, or most modern, if we use the word "modern" according to its dictionary definition.

Re: Redo: A recursive, general-purpose build system

#64

Post by [deleted] »

woodruffw wrote: post #18
AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

Not just a functional language -- I found that my ability to write maintainable (and performance) Makefiles drastically improved when I began to think of Make as a Prolog/Datalog variant with particularly heinous syntax and very limited constraint resolution.

[deleted]

Re: Redo: A recursive, general-purpose build system

#65

Post by ljm »

woodruffw wrote: post #18
AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

Not just a functional language -- I found that my ability to write maintainable (and performance) Makefiles drastically improved when I began to think of Make as a Prolog/Datalog variant with particularly heinous syntax and very limited constraint resolution.

It's funny you mention that because I've wondered for a while what it would be like to have a Prolog-based build system (or even just as a CI config target instead of abusing yaml). I just lack the technical nous (and time) to try and pull it off.

Re: Redo: A recursive, general-purpose build system

#66

Post by lewisl9029 »

lolski wrote: post #15
AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

I second Bazel. People keep on mentioning how steep the learning curve is, but the conceptual model is really simple, elegant, and intuitive. What is steep is the technical know-hows: 1. When things don't work as expected. For example, while it worked flawlessly with languages that it natively support such as Java, that wasn't the case for other languages such as Javascript or Python. 2. When you have to do something…

I think many of the ideas/techniques behind Bazel are excellent: hermiticity, distributed caching, dependency analysis, massively parallel execution, etc. There are so many things every other build system out there can learn and apply from Bazel that could make them 10x more effective.

But IMHO Bazel's fundamental flaw is that it tries to be a one size fits all solution that's supposed to work for all languages and ecosystems. That pretty much guarantees that it will have more need for boilerplate/configuration, be less performant, have more bugs, and at the end of the day just offer a worse user experience than a build system that applies all of Bazel's underlying ideas/techniques while being designed solely for a single language/ecosystem and more deeply integrated into it.

I think the eventual best case scenario for Bazel is a jack of all trades, master of none, but in its current state, for most of the languages/ecosystems it claims to support, it would be way too generous to even call it a jack of the trade in terms of overall UX.

Re: Redo: A recursive, general-purpose build system

#67

Post by pdimitar »

Too wrote: post #54
pdimitar wrote: post #2

Not impressed by shell incantations. What would sell such a tool to me is a feature to replace those with new and more intuitive syntax. And a terser one, too. Holding on to how things are done in the shell is not a thing to be proud of. I think a lot of us around here stopped counting the times we got tripped by globbing, forgetting or misplacing one special character in a ${} block, or quoting. Let those monstrosit…

Did you try https://pydoit.org/ , it's a build tool in similar spirit, which allows the actions to be either python-functions or external programs. Try to look past their tutorial 1, I don't know why they mixed in module_imports there, making it look a lot more complicated than what it really is.

Nice idea, I like it. I have no use for Python functions however, and the startup time of the interpreter will likely get in the way since I need a tool I can invoke many times in scripts as well.

Will try it for sure though, thank you for the link.

Re: Redo: A recursive, general-purpose build system

#68

Post by fivea »

throwaway894345 wrote: post #23
packetlost wrote: post #19

Earlier quoted context omitted.

Bazel is awful unless you're Google and have a team of devs supporting Bazel.

I suspect the experience varies pretty widely based on what languages you use and what you’re trying to do with it. If you’re just building Go binaries then I’m sure Bazel is great, but if you’re doing anything with Python 3 or if you’re doing something a little off the beaten path like code generation, it’s probably a frustrating experience.

> I suspect the experience varies pretty widely based on what languages you use and what you’re trying to do with it.

I second GP's assessment that Bazel is pretty developer-hostile if you are not Google or have Google's resources to maintain Google's build systems.

Case in point: get Bazel to consume system libraries or vended/third-party dependencies in a C++ project. This is a pretty basic usecase, but it was left as an afterthought in Bazel, and it's an uphill battle just to get Bazel to work with them.

I don't doubt that Bazel can be usable if your entire world is locked tight in repos you control and were all forced to migrate to Bazel. It's the same thing with GYP, another developer-hostile build system spawned there. However, that is not how the world outside of Google works.

The only way I see Bazel become relevant and usable for non-Google/FANG-size orgs is if a higher level build system like CMake supports generating Bazel code, so that all these shortcomings are brushed under the proverbial rug. However, if we get to that, are we really using Bazel, or is Bazel relegated to an implementation detail?

Re: Redo: A recursive, general-purpose build system

#69

Post by Arch-TK »

redleader55 wrote: post #46

File system time stamps are an unreliable way to decide if a target needs to be rebuilt. It doesn't work with distributed builds, it doesn't allow instant rebuilds to help with bisections, various file systems might not have not store the same time attributes, even on the same OS, etc. In my experience (large, mono-repo builds) hashes are the way to go and offer the most portable way to map an artefact to a tree of d…

Using filesystem timestamps is a misfeature of THIS particular redo implementation. Almost all the other redo implementations I know of (which are generally also much faster than apenwarr's redo) use a hash if the timestamp has changed.

Also, redo only lets a given .do file produce ONE output from a set of specified inputs. If you want to enforce this, nothing stops you from writing some declarative tool on top of redo (meaning you would replace the shebangs) which implements the sandboxing. Other than that, it's outside of the scope of concerns of a tool such as redo, and adding it would be a misfeature in my opinion (since it's not directly related to handling of the dependency graph).

Re: Redo: A recursive, general-purpose build system

#70

Post by Arch-TK »

floatboth wrote: post #35

Just like tup, redo keeps popping up as a nice conceptual thing, but none of these take off in practice. For most projects the winning build system is not the most conceptually interesting one, nor the most flexible one. It's the most "convention over configuration" one. This is why Meson has taken the freedesktop/gnome/etc world by storm.

It's not hard to "take over the freedesktop/gnome" world by storm since it's basically the same small group of people who have any power in that world.
Post reply on HN