Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redo: A recursive, general-purpose build system

redo.readthedocs.io

Post reply on HN
Print view View original

51–60 of 95 posts

Re: Redo: A recursive, general-purpose build system

#51

Post by Nzen »

redleader55 wrote: post #46

File system time stamps are an unreliable way to decide if a target needs to be rebuilt. It doesn't work with distributed builds, it doesn't allow instant rebuilds to help with bisections, various file systems might not have not store the same time attributes, even on the same OS, etc. In my experience (large, mono-repo builds) hashes are the way to go and offer the most portable way to map an artefact to a tree of d…

The sandboxing restriction is one that merits care for (or not supporting) some use cases. Tup requires declaring all outputs (but not all inputs), which can be inconvenient when a build process creates intermediate or derived files that are tedious to anticipate. [0]

[0] https://groups.google.com/g/tup-users/c/umW73zR5JKc?pli=1 . ex java creates numbered .class files for anonymous classes declared in a larger java file. In fairness, while looking, they may have relaxed some of that since I last looked with a transient flag https://github.com/gittup/tup/issues/405

Re: Redo: A recursive, general-purpose build system

#52

Post by AceJohnny2 »

einpoklum wrote: post #16
AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

> All the alternative build systems wisely choose to tackle only some of these features. But why should all of these features be tackled by a single piece of software? Especially seeing how Make, in itself, has proven wildly insufficient already... what, 30 years ago, when GNU autotools came onto the scene? More specifically, why do you find combinations such as meson + ninja, or CMake + ninja, or even CMake + make a…

That was exactly my point: perhaps it's best not to bundle all those features in a single piece of software.

I haven't tried all the alternative combinations because, ultimately, I only have so much time and interest, and their learning curve is significant.

Re: Redo: A recursive, general-purpose build system

#53

Post by Too »

thechao wrote: post #26
AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

I skimmed the paper. I use Make, extensively, but in a way I generally don’t see in the wild (for C/++): 1. I convert all paths to abspaths; 2. All intermediate products go into a temp dir; 3. All recipes are functions; 4. The set of objects are defined by a “find . -name ‘…’” for C sources; 5. Every C file depends on every header file in the repo; 6. Use ccache; and, 7. Deterministic build. My big work project is ~2…

> Every C file depends on every header file in the repo.

So change in single header file always becomes a full build? Do you use pimpl or other tricks to avoid frequent changes to h-files, otherwise class members usually end up in the headers for simplicity and to avoid incomplete type problems on non-pointers unfortunately. Was this C or C++?

What's the difference between a clean rebuild and a scratch build?

Re: Redo: A recursive, general-purpose build system

#54

Post by Too »

pdimitar wrote: post #2

Not impressed by shell incantations. What would sell such a tool to me is a feature to replace those with new and more intuitive syntax. And a terser one, too. Holding on to how things are done in the shell is not a thing to be proud of. I think a lot of us around here stopped counting the times we got tripped by globbing, forgetting or misplacing one special character in a ${} block, or quoting. Let those monstrosit…

Did you try https://pydoit.org/, it's a build tool in similar spirit, which allows the actions to be either python-functions or external programs. Try to look past their tutorial 1, I don't know why they mixed in module_imports there, making it look a lot more complicated than what it really is.

Re: Redo: A recursive, general-purpose build system

#55

Post by Too »

floatboth wrote: post #36
lolski wrote: post #15

Earlier quoted context omitted.

I second Bazel. People keep on mentioning how steep the learning curve is, but the conceptual model is really simple, elegant, and intuitive. What is steep is the technical know-hows: 1. When things don't work as expected. For example, while it worked flawlessly with languages that it natively support such as Java, that wasn't the case for other languages such as Javascript or Python. 2. When you have to do something…

Also Bazel is very "corporate", it's not designed for projects that want to be good FOSS unix citizens. Very telling: "common c++ use cases" https://docs.bazel.build/versions/4.2.2/cpp-use-cases.html includes pulling googletest source from the web and linking a random shared object blob. Zero mentions of pkg-config.

The http_archive has a sha256-attribute to deal with the "random". You could add multiple urls to point to your local mirror as well. It's a way to pin external dependencies and get reproducible builds without checking in a big blob into your git. pkg-config in my experience is less reproducible and tends to pollute and be affected by system environment, unless you have some build guru who can setup chroot in your project.

Re: Redo: A recursive, general-purpose build system

#56

Post by JdeBP »

sigjuice wrote: post #32
codedokode wrote: post #20

Earlier quoted context omitted.

I don't like using tabs in Makefile syntax. Tabs are in many cases (especially on print) indistinguishable from spaces. Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.

make doesn't really understand C or C dependencies, but the compilers usually do. Here is a minimal example that will take care of dependencies by passing -MD -MP to gcc or clang. The dependencies (*.d files) get generated as part of the normal gcc/clang invocations. There is no separate dependency generation step. $ cat Makefile CFLAGS := -MD -MP hello: hello.o x.o y.o -include *.d $ echo '#include "x.h"' > x.c $ ec…

Actually, the compilers do not and you are doing less than half of the job there. They don't emit information about the non-existent files that would change the build if they were to suddenly exist, such as an "x.h" in one of several standard search directories in that example.

* http://jdebp.uk./FGA/introduction-to-redo.html#CompilerDefic...

Re: Redo: A recursive, general-purpose build system

#57

Post by JdeBP »

joshuamorton wrote: post #41
codedokode wrote: post #20

Earlier quoted context omitted.

I don't like using tabs in Makefile syntax. Tabs are in many cases (especially on print) indistinguishable from spaces. Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it.

> Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it. This requires that either you implement or shell out to a C/C++ parser as part of the dependency resolution step (which has to happen prior to the actual compilation). This will slow down your compilation a lot unless you're able to parse C/…

The idea behind redo, in contrast, is that "dependency resolution" is not a separate step. The dependencies can be generated at the same time as the compilation, by the compilation in a perfect world; through some post-processing in a less than perfect one.

Unfortunately, current C and C++ compilers do not emit all of the information that they possess internally, only less than half of it (the positive information about files found, not the larger amount of negative information about non-existent files that were searched and would change the build if they were to exist).

* http://jdebp.uk./FGA/introduction-to-redo.html

Re: Redo: A recursive, general-purpose build system

#58

Post by joshuamorton »

JdeBP wrote: post #57
joshuamorton wrote: post #41

Earlier quoted context omitted.

> Also, any "make" program for C should automatically detect dependencies between files because this information is present in the code and it doesn't make sense to duplicate it. This requires that either you implement or shell out to a C/C++ parser as part of the dependency resolution step (which has to happen prior to the actual compilation). This will slow down your compilation a lot unless you're able to parse C/…

The idea behind redo, in contrast, is that "dependency resolution" is not a separate step. The dependencies can be generated at the same time as the compilation, by the compilation in a perfect world; through some post-processing in a less than perfect one. Unfortunately, current C and C++ compilers do not emit all of the information that they possess internally, only less than half of it (the positive information ab…

I guess my point is that in the case of some target a that depends on some target b existing (I'm not sure what the best example is here, I know that bazel takes advantage of this to parallelize compilation, but I'm not intimately familiar with c++ compilation internals, so it might be clearer with a generated file example).

You have a.cc that depends on b.cc, but some tool generates b.cc, so it may not exist (but `b.do` does). If I invoke `a.do` to compile `a.cc`, and don't directly declare b.cc as a dependency in the .do file, the theoretical compiler will resolve that it depends on b.cc, and note that that doesn't exist, and then what?

In the bazel world, since you define the dependency graph explicitly before invoking any compiler, you can't run into this issue, but in situations where you need to get dep info from the compiler...what happens with stuff like this?

Re: Redo: A recursive, general-purpose build system

#59

Post by hv42 »

AceJohnny2 wrote: post #4

As an intensive (and reluctant) user of GNU Make, I keep looking for a more modern replacement. Redo is not it. Make is really a complicated combination of these components: - a dependency definition language with in-place evaluation and half-assed wildcards - A functional language with extremely limited datatypes (effectively just space-separated strings) - a clunky macro system - A "distributed" topological executi…

Gradle is a good make replacement as well. The kotlin DSL is much more readable and easier to follow than makefiles and has a lot of additional features.

Re: Redo: A recursive, general-purpose build system

#60

Post by floatboth »

Too wrote: post #55
floatboth wrote: post #36

Earlier quoted context omitted.

Also Bazel is very "corporate", it's not designed for projects that want to be good FOSS unix citizens. Very telling: "common c++ use cases" https://docs.bazel.build/versions/4.2.2/cpp-use-cases.html includes pulling googletest source from the web and linking a random shared object blob. Zero mentions of pkg-config.

The http_archive has a sha256-attribute to deal with the "random". You could add multiple urls to point to your local mirror as well. It's a way to pin external dependencies and get reproducible builds without checking in a big blob into your git. pkg-config in my experience is less reproducible and tends to pollute and be affected by system environment, unless you have some build guru who can setup chroot in your pr…

Oh I didn't mean "random" as in "could change". In fact "random" was about the unspecified .so blob, not the http archive.

> pkg-config in my experience is less reproducible

But this is what I mean, you only want full reproducibility in a "corporate" environment. In a FOSS desktop environment you often specifically want to use whatever the system has.

Post reply on HN