Redo: A recursive, general-purpose build system

Part of the language builtin to my build system provides ways of running commands without a shell.

> there is no gcc or a compiler which takes gcc-like options,

No, but there is MSVC. You just need to abstract the compiler and compiler flags. CMake did it, so why can't I?

> there's no posix compliant or gnu compliant make.

I don't need a make, and neither should redo. My build system will not be like CMake in that it will generate build files for something else; it will execute them itself. No make necessary. In fact, I could implement a make easily with my build system; just a parser would be necessary.

> If you plan on making a build system, it has to take a lot of things and provide an abstraction layer over them to make them work for windows. I say forget it, it's a waste of effort, just make it work on linux and by extension (through WSL or mingw or whatever) you get windows.

I don't think it's a waste of effort. People on Windows deserve a good build system too. In fact, not having a good one means that they are dependent on Microsoft. I'd like to help them reduce that dependence.

> One of the best tests of a build system I've found has been to get a build system to build an executable and run it to produce an output which is then consumed as part of the build process. With pure make, this is purely awful and requires either generating a bunch of rules or lots of boilerplate, and generally it's easy to get wrong. With redo I got this working with relatively little work and got it to work with cross-builds.

It's not hard to get make to do that. See [1] and [2].

> In my adventures with redo, the biggest obstacle to hermetic builds that I've found aside from just the environment (env vars) has been compilers.

Yes, but however it's done, it's still easiest in a self-contained system, IMO.

> Surely a better build system would need some form of overhaul to the interfaces provided by compilers? (Or, some re-implementation of some compiler features so you can spoon-feed a compiler the exact flags to ensure it behaves predictably even if the environment changes.)

The build system should carefully control the environment. In fact, it should make sure the environment is empty. (There are details, but in general...) That's the biggest thing to make sure the compiler does not act weird.

You should also do the build in a separate directory and copy needed files into the directory. Then check all commands and their arguments for possible files before running them. This is one reason why depending on an outside shell is not ideal.

Hermetic builds won't ever be perfect, but you can make it 80% of the way with only a few things.

> TFA?

"The Famous Article". It refers to the link under discussion on HN. In this case, it refers to the apenwarr redo.

> Okay, so you mean dynamically generated .do files.

Not just that. Those were just an example of how dynamic dependencies are already awkward in redo. There are countless other uses for dynamic dependencies. To give you a taste, my build system will actually be able to build a distributed system, i.e., it will be able to be used for DevOps deployment, and it will be easy.

Surely redo can as well, but as I've thought about how to do that, it seems awkward. Perhaps you can prove me wrong?

> Why finnicky? In the usual case you DON'T want anything other than just timestamps and hashes, it works for all cases, albeit slowly for some. If you NEED to customize it, you have to pull out the special tools but then you just wrap that in some little utility and mostly forget about it.

Finicky because as far as I can tell, the build database in redo is modified by multiple commands over a build. This can cause problems should a problem occur with the database mid-build. In a self-contained system, the database is in-memory during the entire build and only written out to disk once the build is done, which can then be used as a sort of "commit" of the build, marking it as finished.

There are other reasons, but that's the first that comes to mind.

> What do YOU think are the obstacles to hermetic builds? I already know what I've found but I'm always looking for more input.

I think a good paper to read would be the Dolstra thesis on Nix. [3] If you consider Nix a build system (not just a package manager), it goes further than even Bazel. The thesis should be a good place to start.

[1]: https://git.yzena.com/gavin/bc/src/branch/master/Makefile.in...

[2]: https://git.yzena.com/gavin/bc/src/branch/master/configure.s...

[3]: https://edolstra.github.io/pubs/phd-thesis.pdf

I didn't meant to imply that you or redo needs a make. Just that for something like autotools or cmake, there's a lot of abstraction required just to get things to work. Or you can bundle things or you can write your own things but the point is, it's a lot of work.

>I don't think it's a waste of effort. People on Windows deserve a good build system too. In fact, not having a good one means that they are dependent on Microsoft. I'd like to help them reduce that dependence.

Right, but as I point out, microsoft already took care of that by giving windows users WSL.

>It's not hard to get make to do that. See [1] and [2].

Yes I know what it takes, but if you don't think this is horrifically over-complicated then I don't know what to tell you. Instead of using implicit rules with target specific overrides you've made it slightly less boilerplaty by making it less scalable and expanding the entire build command for a specific target. I still think this is too much boilerplate and noise.

>Yes, but however it's done, it's still easiest in a self-contained system, IMO.

I think it may be easiest to implement (just like any complicated solution is easier to implement than a simple one) but I don't think it's a good goal for a build system. Complexity isn't free, even if it has a cheaper initial cost.

>The build system should carefully control the environment. In fact, it should make sure the environment is empty. (There are details, but in general...) That's the biggest thing to make sure the compiler does not act weird.

I disagree, you would also need to compile the compiler (because compilers like to take compile-time configuration). Moreover, you're going to have to have specific code for handling every different compiler with all its quirks, you're also going to have to spend a lot of time figuring out all the quirks. And you're going to have to spend a lot of time duplicating the code which implements those quirks.

Surely getting compilers to support some standard hermetic building facilities would be easier and better in the long run than maintaining a complex database of compiler quirks to handle for every specific compiler version etc?

> Hermetic builds won't ever be perfect, but you can make it 80% of the way with only a few things.

Yes but I can already get it 80% of the way with redo.

> Surely redo can as well, but as I've thought about how to do that, it seems awkward. Perhaps you can prove me wrong?

I'm still not sure we're both on the same page for what a "dynamic dependency" is but with redo I've solved the problem of having a dynamic .do file by simply depending on a dynamically generated .rc file: i.e a file containing sourcable shell variables (including bash arrays), generated using declare -p or things like the @Q parameter transformation.

>Finicky because as far as I can tell, the build database in redo is modified by multiple commands over a build. This can cause problems should a problem occur with the database mid-build. In a self-contained system, the database is in-memory during the entire build and only written out to disk once the build is done, which can then be used as a sort of "commit" of the build, marking it as finished.

There are no concurrency issues with the build database in any redo implementation worth it's salt and realistically there's nothing stopping a redo implementation from implementing its database the way you describe. It's also an implementation detail which I've never had to interact with. Not sure how that makes it finnicky.

> I think a good paper to read would be the Dolstra thesis on Nix. [3] If you consider Nix a build system (not just a package manager), it goes further than even Bazel. The thesis should be a good place to start.

From what I've seen, nix is too coarse grained for being a build system, which is where I've seen bezel recommended, but both of these fail to provide a general purpose build system which can be used outside a tightly controlled environment to build a piece of software hermetically using existing system dependencies. At the end of the day, I want to be able to build software with existing system dependencies provided by my package manager with the assurance that if I hit build, if those system dependencies have changed, the relevant parts of my code will be forced to rebuild and I will be told about it. I can see the value of totally hermetic builds, but a build system capable of solving the problem I describe should easily be extendable to solve the general problem of hermetic builds. In short, I want to be able to compile my code with my existing compiler, and if my compiler is updated, I want to be able to hit redo and have everything rebuilt correctly without worrying that something didn't get built.

I will say, though, that if you're not hearing about my build system in a few years, you can safely assume that you were right and that I was wrong.