Makefiles – Best Practices

I haven't found a reason to not just use Make yet when starting from green fields, including some very big projects. In fact, I have often been annoyed at some projects picking something like CMake for no reason beyond it's "more advanced," but which ends up just being an extra dependency I have to fetch and install.

If I were to pick one of the above for building large projects on linux variants, which one would you pick?

For me, I mostly work in embedded Linux - that's Linux for routers, custom hardware, sometimes a Docker image, etc. That means lots of packaging work. Sometimes it means build machines that can't even run the compiled binary. I compile libraries for platforms that the original author would never have imagined.

In that space, there are lots and lots of details that need to be "just so". Lots of specifics about the cflags and linker flags. Requirements on where 'make install' puts files. Sometimes restrictions where an output binary wants to search for config files (spoiler: use sysconfdir).

Out of all the libraries/programs that I've had to package, autotoolized projects are the easiest to handle by far, followed by CMake projects.

Hand-written Makefiles are usually painful to a packager in one way or another. The most common ones being hard-coded CC/CXX/CFLAGS/CXXFLAGS, non-standard target names, and non-standard/incomplete usage of directory variables.

I know that Autotools is crusty in a lot of ways, and that the learning curve is steep. It's a nasty mix of Perl and m4, and runs lots of checks that don't matter at all. It's also what I use for all of my own libraries and programs, because the result is worth the pain (for me).

So for any program that I expect more than two humans to ever compile, I'd recommend Autotools if you're a perfectionist and CMake if you're not. Within Autotools, I'd recommend Automake and Autoconf if the language is C/C++, and just Autoconf/Make otherwise. (But be sure to follow the Makefile conventions: https://www.gnu.org/software/make/manual/html_node/Makefile-...).

I wouldn't recommend a hand-rolled Makefile unless you're the only one using it, the build is really weird for some reason, or unless it's a wrapper around some other build tool.

Thanks, I bought your GNU Make book a couple years ago and read pretty much the whole thing! Along with reading the GNU Make manual a few years before that, I made an attempt to "give Make a fair shake".

I liked it, but it's honestly weird to me that a book published in 2015 can improve on the state of the understanding of a tool with 40 years of heritage :) I also get this "groundhog day" effect after about 10 years of seeing new GNU make tutorials / best practices on Hacker News every couple months. Everybody who learns Make has to go through this same thing.

-----

I read that you reimplemented GNU Make for Electric Cloud and I was impressed by that :) My project Oil [1] is a similar sort of project. It can run thousands of lines of unmodified shell/bash scripts found "in the wild". I got big distro scripts working last year, and I got thousands of lines of interactive completion scripts working recently, which I need to blog about.

For awhile I thought it would be nice to replace GNU make too, although (1) that's a lot of effort, and following Oil's strategy isn't possible since Makefiles can't be statically parsed and (2) I think it's happening anyway.

Major build systems are now split up into high-level and low-level parts, i.e. autoconf generating Makefiles, CMake generating Makefiles/ninja files. Android used to be 250K lines of pure GNU make (including GMSL), but now it's a high level Blueprint DSL generating Ninja too.

There aren't many pure Make projects anymore, at least in open source code. The exception I can think of are embedded ones like buildroot.

I like how Ninja focuses on build execution only, punting logic to a higher level (CMake, gyp, Blueprint). So my pet theory is that you can replace Make with a DSL that generates

1) A ninja file for fast, parallel, incremental developer builds

2) A shell script for portable builds for distro packagers/end users. This is just a clean serial build, so it can be a shell script rather than a Makefile.

I plan to test this theory by throwing out the modest 700 lines of Make I've written from scratch and replacing it with a Ninja/shell generator :)

I've debugged and read enough Make to be able to identify and fix most problems. But it still feels like like whack-a-mole to me. You can fix one problem and introduce another, since there's no real way to test for correctness (and efficiency). Problems can be reintroduced by seemingly innocuous changes.

[1] http://www.oilshell.org/blog/2018/01/28.html

+1 for a great book, I bought that book too!

I use the patterns in @jgrahamc's article "Escaping: A Walk on the Wild Side" pretty often. Especially the \n definition, which works great to put on the end of a $(foreach) loop inside of a recipe.

@jgrahamc: Thanks for all you've written on Make over the years.

I think the main point nowadays is how to write Makefiles for highly parallel builds. It's surely the thing that gave me the most headaches. In the end such a talk will probably mostly be about how to avoid calling Make recursively. Another talk I'd like to hear, which would be more meta: does it still make sense to bother with Make at all, aside from maintaining legacy systems? Are there features or paradigms which still make it worthwhile, compared to other popular build systems?