Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redo: A recursive, general-purpose build system

redo.readthedocs.io

Post reply on HN
Print view View original

71–80 of 95 posts

Re: Redo: A recursive, general-purpose build system

#71

Post by Arch-TK »

gavinhoward wrote: post #40

I feel like DJB's ideas behind redo are actually some of his most poorly thought-out ideas. It seems that he thought, 'Make has . To solve , a new build system should have .' In other words, he seems to just used additive solutions instead of subtractive ones. [1] Most build systems seem to be like that, adding features on top of features. The end result is usually complicated and finicky. I think that's why build sy…

Aside from the fact that DJB barely specified redo to the point where you could make accusations claiming it did or did not do something, I think there are implementations of redo which are capable of fitting the requirements you specify.

1. Redo doesn't give you the tools to have dynamic dependencies for example, but it also doesn't concern itself with those and doesn't prevent you from having them. I have a small project which has dynamic dependencies (not system dependencies but just configuration options) and handles them gracefully with redo (without ever needing a make clean).

2. Flexible file stamps are something which you can implement on some redo implementation using a combination of redo-always and redo-stamp.

3. The targets and build scripts being turing complete is something which redo doesn't specify but that is the intentional use of redo.

That being said, I think there is a place for something even more BASIC than redo which can be used more directly to implement what you've described while also being flexible enough to build redo on top of. It has given me things to think about.

Re: Redo: A recursive, general-purpose build system

#72

Post by Arch-TK »

stargrave wrote: post #13

One of redo's implementations (on Go) has complete documentation on the whole redo build system (applicable to most (all?) redo-s) usage: http://www.goredo.cypherpunks.ru/Usage-rules.html

Not all redo implementations treat $2 the same. Specifically JDEBP's redo differs in that $2 is the extension rather than the part with the extension removed (it's unspecified what $2 is for non default*.do files).

Re: Redo: A recursive, general-purpose build system

#73

Post by brandmeyer »

glandium wrote: post #61
JdeBP wrote: post #56

Earlier quoted context omitted.

Actually, the compilers do not and you are doing less than half of the job there. They don't emit information about the non-existent files that would change the build if they were to suddenly exist, such as an "x.h" in one of several standard search directories in that example. * http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDefic...

Also they don't add an empty rule for the source file (like they can do for headers), so if you rename from e.g. foo.c to foo.cpp, which still produces a foo.o, your incremental build fails because the dep file still has the dep on foo.c which doesn't exist anymore.

The original sin here is generating the dependencies from the source files in the same step that was used to generate the object files. The only way to ensure that Make observes structural changes in the dependency DAG is if the graph is rebuilt.

Re: Redo: A recursive, general-purpose build system

#74

Post by remram »

codedokode wrote: post #20
AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

I don't like using tabs in Makefile syntax. Tabs are in many cases (especially on print) indistinguishable from spaces. Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.

Why do you need to distinguish tabs from spaces?

Re: Redo: A recursive, general-purpose build system

#75

Post by thechao »

Too wrote: post #53
thechao wrote: post #26

Earlier quoted context omitted.

I skimmed the paper. I use Make, extensively, but in a way I generally don’t see in the wild (for C/++): 1. I convert all paths to abspaths; 2. All intermediate products go into a temp dir; 3. All recipes are functions; 4. The set of objects are defined by a “find . -name ‘…’” for C sources; 5. Every C file depends on every header file in the repo; 6. Use ccache; and, 7. Deterministic build. My big work project is ~2…

> Every C file depends on every header file in the repo. So change in single header file always becomes a full build? Do you use pimpl or other tricks to avoid frequent changes to h-files, otherwise class members usually end up in the headers for simplicity and to avoid incomplete type problems on non-pointers unfortunately. Was this C or C++? What's the difference between a clean rebuild and a scratch build?

Yes: touching any header file causes a full rebuild. Remember that ccache is "under the hood" doing memoization. So, a clean rebuild could still hit ccache, whereas a scratch build starts from an emoty ccache.

Re: Redo: A recursive, general-purpose build system

#76

Post by ghoward »

Arch-TK wrote: post #71
gavinhoward wrote: post #40

I feel like DJB's ideas behind redo are actually some of his most poorly thought-out ideas. It seems that he thought, 'Make has . To solve , a new build system should have .' In other words, he seems to just used additive solutions instead of subtractive ones. [1] Most build systems seem to be like that, adding features on top of features. The end result is usually complicated and finicky. I think that's why build sy…

Aside from the fact that DJB barely specified redo to the point where you could make accusations claiming it did or did not do something, I think there are implementations of redo which are capable of fitting the requirements you specify. 1. Redo doesn't give you the tools to have dynamic dependencies for example, but it also doesn't concern itself with those and doesn't prevent you from having them. I have a small p…

> Aside from the fact that DJB barely specified redo to the point where you could make accusations claiming it did or did not do something, I think there are implementations of redo which are capable of fitting the requirements you specify.

You're not wrong, and I think DJB did get there. However, there's a difference between the Unix way of designing a system out of disparate pieces working together (something DJB is good at and tends to do) and designing a cohesive whole system.

Obviously, redo is the former. My build system will be the latter, and I am doing it that way (despite being a fan of the Unix way in general) in an attempt to 1) hopefully make a build system that the masses actually like, 2) make it easier to make it work on Windows from the start, and 3) a cohesive whole build system is a much better platform for implementing hermetic builds.

(There are a bunch of smaller reasons, such as separating file targets and non-file targets, but all of those reasons are details.)

So yes, you are absolutely correct that redo can do everything I mentioned, and I guess that's an L for me. I should have mentioned other features as well.

> 1. Redo doesn't give you the tools to have dynamic dependencies for example, but it also doesn't concern itself with those and doesn't prevent you from having them. I have a small project which has dynamic dependencies (not system dependencies but just configuration options) and handles them gracefully with redo (without ever needing a make clean).

That's not quite what I envision when I say "dynamic dependencies" (my build system will also have configuration, but the config changing will not use dynamic dependencies), but yes.

However, at least in the TFA redo, dynamic dependencies (or simulating them) can be awkward. [1]

> 2. Flexible file stamps are something which you can implement on some redo implementation using a combination of redo-always and redo-stamp.

Correct. And I should have mentioned that. But again, DJB did that in an indirect way, making the system "finicky" (in my opinion).

> 3. The targets and build scripts being turing complete is something which redo doesn't specify but that is the intentional use of redo.

Yeah, no argument there.

> That being said, I think there is a place for something even more BASIC than redo which can be used more directly to implement what you've described while also being flexible enough to build redo on top of. It has given me things to think about.

Well, I don't think going even more basic is good here. Hermetic builds are just too good to pass up, and that requires some complexity.

[1]: https://redo.readthedocs.io/en/latest/FAQImpl/#can-a-do-file...

Re: Redo: A recursive, general-purpose build system

#77

Post by kaba0 »

dotancohen wrote: post #62
kylecordes wrote: post #43

Earlier quoted context omitted.

Of course that is an overstatement, but: I teach Bazel and help companies adopt it. I’ve noticed that a high percentage of those using Bazel have one or more xooglers on their team.

> I teach Bazel and help companies adopt it. The very fact that this occupation exists pushes me away from Bazel, having never used it. Makefiles are horrible, but they are well documented and I've been able to google my way out of every problem that I've either run into, inadvertently caused, or had to debug from somebody else's mess (it's always a mess). I don't actually want a better build tool. I want a tool that…

But that’s not really how it works. Build systems exist in a complex domain — whether we like it or not, that complexity can’t really be hidden away, it will leak one way or another.

Re: Redo: A recursive, general-purpose build system

#78

Post by seanstrom »

w4rh4wk5 wrote: post #29
seanstrom wrote: post #22

Please someone mention Tup and how it can be a reasonable build system. I’ve heard the Fuse dependency is not ideal, though I felt it had a nice UX experience with the Lua config. Plus it’s worth considering that you can potentially use Fennel to configure the builds (since Fennel compiles to Lua). Tup: https://github.com/gittup/tup Fennel: https://fennel-lang.org/

I have encountered Tup in the past and could not figure out how to define a "generator". As in: a function that defines how some output can be generated from a certain input running multiple commands. I don't want to copy those commands for every input I need to process. Edit: Generator is a term typically used in CMake and Meson for this, in Make I'd use a pattern rule mostly.

Not sure how to exactly to build a "generator" either, but it seems that it would be a build rule that generates multiple outputs from multiple inputs right? If that's the case there's a `foreach` function for convenience, but it doesn't seem to have something for multiple commands. Though there's this Lua example on their website that leaves me wondering if what you want is possible, here's the code (since the documentation seems offline atm):

```

inputs = { 'file1.c', 'file2.c' }

outputs = { '%B.o' }

commandA = 'gcc %f -c -o %o'

commandB = 'gcc %f -o %o'

objects = tup.foreach_rule(inputs, commandA, outputs) tup.rule(objects, commandB, {'app'})

```

Which apparently is a shortcut for saying:

```

tup.definerule{

    inputs = {'file1.c'},  
    command = 'gcc file1.c -c -o file1.o',  
    outputs = {'file1.o'}
}

tup.definerule{

    inputs = {'file2.c'},  
    command = 'gcc file2.c -c -o file2.o',  
    outputs = {'file2.o'}
}

tup.definerule{

    inputs = {'file1.o', 'file2.o'},  
    command = 'gcc file1.o file2.o -c -o app',  
    outputs = {'app'}
}

```

Reference: http://web.archive.org/web/20201026140926/http://gittup.org/...

Re: Redo: A recursive, general-purpose build system

#79

Post by sigjuice »

JdeBP wrote: post #56
sigjuice wrote: post #32

Earlier quoted context omitted.

make doesn't really understand C or C dependencies, but the compilers usually do. Here is a minimal example that will take care of dependencies by passing -MD -MP to gcc or clang. The dependencies (*.d files) get generated as part of the normal gcc/clang invocations. There is no separate dependency generation step. $ cat Makefile CFLAGS := -MD -MP hello: hello.o x.o y.o -include *.d $ echo '#include "x.h"' > x.c $ ec…

Actually, the compilers do not and you are doing less than half of the job there. They don't emit information about the non-existent files that would change the build if they were to suddenly exist, such as an "x.h" in one of several standard search directories in that example. * http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDefic...

If a /usr/include/x.h were to appear, wouldn't that have lower priority than the x.h in the current directory and therefore not change the build?

Re: Redo: A recursive, general-purpose build system

#80

Post by glandium »

brandmeyer wrote: post #73
glandium wrote: post #61

Earlier quoted context omitted.

Also they don't add an empty rule for the source file (like they can do for headers), so if you rename from e.g. foo.c to foo.cpp, which still produces a foo.o, your incremental build fails because the dep file still has the dep on foo.c which doesn't exist anymore.

The original sin here is generating the dependencies from the source files in the same step that was used to generate the object files. The only way to ensure that Make observes structural changes in the dependency DAG is if the graph is rebuilt.

Good old "make depend". If only it didn't add a significant overhead...
Post reply on HN