Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Makefiles – Best Practices

danyspin97.org

Post reply on HN
Print view View original

101–110 of 127 posts

Re: Makefiles – Best Practices

#101

Post by mturmon »

kajm wrote: post #100
nrclark wrote: post #40

GNU Make is an awesome task-runner. I use it all the time, for all sorts of things. It's my shell-script replacement, and I write one-off Makefiles frequently. It's got a nice macro-processor, some helpful string manipulation tools, and expresses task dependencies perfectly. The manual is also very well written. But if you're writing a software package, consider using CMake, Meson, Autotools, or something similar. Un…

> awesome task-runner I do this too. I have been trying to find examples of people doing something similar on the web but had no luck. I've replaced so many 5-line shell scripts in my user /bin by just having a single makefile in the root of my home directory and having aliases to the various commands defined in the makefile. I even have a rule that shows a rofi menu that allows me to run the rules easily although it…

Thirded - in a slightly different context. (See also: https://news.ycombinator.com/item?id=5275313#5276744)

I’m currently managing data reduction for a bunch of space mission simulations using make. A simulation dumps a bunch of files as python pickles, which are reduced into csv, and then made into plots, and then into webpages. Each phase of this is in the makefile. None of the transformation rules are standard compilation macros like you expect to find in a makefile.

The setup makes it easy to update all the summary webpages when more sims are run. Just “make html”. It re-accumulates all the summary metrics, redoes graphics, remakes webpages.

This is the kind of processing pipeline that is often done in shell or with driver scripts, but then you always have to remake everything. I like the setup a lot.

Re: Makefiles – Best Practices

#102

Post by Samis2001 »

kazinator wrote: post #89
chubot wrote: post #25

Earlier quoted context omitted.

FWIW I also read the GNU Make manual, and based some code for automatic deps off a profoundly ugly example it had. Then later people on HN showed me a better/simpler way to do it. https://news.ycombinator.com/item?id=15060149 https://www.gnu.org/software/make/manual/html_node/Automatic... After reading the manual and writing 3 substantial Makefiles from scratch, I still think Make is ugly and, by modern standards, no…

> offers you virtually no help in writing correct parallel and incremental builds. The key problem there that Make has no idea about the semantics of the shell code that appears in the build recipes. It has no idea how two build recipes interact with each other through side effects on objects that are not listed as targets or prerequisites. I think ClearCase's clearmake (GNU-compatible) actually intercepts the file s…

There have been systems that take this approach such as fabricate.py (https://github.com/brushtechnology/fabricate) and tup (http://gittup.org/tup/).

Re: Makefiles – Best Practices

#103

Post by mturmon »

AceJohnny2 wrote: post #29
jgrahamc wrote: post #17

Earlier quoted context omitted.

https://www.cmcrossroads.com/article/tips-and-tricks-automat...

Thanks John, your book is the most-referred to on my office shelf ;) Edit: https://nostarch.com/gnumake (via https://blog.jgc.org/2015/04/the-gnu-make-book-probably-more... )

I also recommend this book. The series on “meta programming make” is also quite good: http://make.mad-scientist.net/category/metaprogramming/

Re: Makefiles – Best Practices

#104

Post by nrclark »

mturmon wrote: post #101
kajm wrote: post #100

Earlier quoted context omitted.

> awesome task-runner I do this too. I have been trying to find examples of people doing something similar on the web but had no luck. I've replaced so many 5-line shell scripts in my user /bin by just having a single makefile in the root of my home directory and having aliases to the various commands defined in the makefile. I even have a rule that shows a rofi menu that allows me to run the rules easily although it…

Thirded - in a slightly different context. (See also: https://news.ycombinator.com/item?id=5275313#5276744 ) I’m currently managing data reduction for a bunch of space mission simulations using make. A simulation dumps a bunch of files as python pickles, which are reduced into csv, and then made into plots, and then into webpages. Each phase of this is in the makefile. None of the transformation rules are standard co…

I do the exact same thing with oddball compilation tasks like this, such as compiling a Latex document or batch-processing pictures.

I also use it to run code-formatters / static analysis tools while programming, sometimes with an secondary Makefile that has a name like 'maintainer.mk'.

Re: Makefiles – Best Practices

#105

Post by self_awareness »

q3k wrote: post #86
self_awareness wrote: post #50

Earlier quoted context omitted.

Users of SCons ( https://scons.org/ ), and Waf ( https://waf.io/ ) disagree with your point of view.

Incidentally, I have not ever met a single user of either of those that does not hate working with them.

Maybe they're simply not from your social circle?

Re: Makefiles – Best Practices

#106

Post by solatic »

Make is a very underappreciated tool outside C/C++ circles.

I'm currently using it in an environment which uses Concourse as the CI tooling - Concourse takes care of version and dependency management at its level, and Make fills in the gaps for fetching the latest versions for local development environments. Because Make won't re-download files for already-made targets, local build cycles are fast after initial setup. Concourse can then re-use these Makefiles by symlinking the resources it manages before calling make, and Make won't try to re-download the resources. If you're not concerned with portability, you can even use make to fetch (and clean) Docker images since they live in predictable directories on Linux (but not OS X).

More people ought to approach the tool with an open mind.

Re: Makefiles – Best Practices

#107

Post by jgrahamc »

wahern wrote: post #60
jgrahamc wrote: post #22

Earlier quoted context omitted.

Both $ cat Makefile $(info FOO $(flavor FOO)) FOO ?= foo $(info FOO $(flavor FOO)) BAR := $(info BAR $(flavor BAR)) BAR ?= bar $(info BAR $(flavor BAR)) BAZ = $(info BAZ $(flavor BAZ)) BAZ ?= baz $(info BAZ $(flavor BAZ)) $ make FOO undefined FOO recursive BAR simple BAR simple BAZ recursive BAZ recursive If undefined then becomes recursive, otherwise the flavour is preserved.

Make variable expansions are always recursive. For what you're describing I think the proper term as used by the GNU Manual is deferred. In other languages the phrase lazy evaluation is common. The one thing I've always had trouble remembering with GNU Make is precedence. Variables defined as command-line arguments (make FOO=bar) override assignments, but environment variables (FOO=bar make) only override ?= assignme…

The GNU Make manual defines two variable types: recursively expanded and simple (see: https://www.gnu.org/software/make/manual/html_node/Flavors.h...).

Re: Makefiles – Best Practices

#108

Post by jgrahamc »

mturmon wrote: post #103
AceJohnny2 wrote: post #29

Earlier quoted context omitted.

Thanks John, your book is the most-referred to on my office shelf ;) Edit: https://nostarch.com/gnumake (via https://blog.jgc.org/2015/04/the-gnu-make-book-probably-more... )

I also recommend this book. The series on “meta programming make” is also quite good: http://make.mad-scientist.net/category/metaprogramming/

Ooh. That's great. I had not seen that!

Re: Makefiles – Best Practices

#109

Post by jgrahamc »

LukeShu wrote: post #69
jgrahamc wrote: post #14

Earlier quoted context omitted.

Yes. It is highly recommended that you read that book (buy it to support the FSF). If you want more I wrote a book on GNU Make ( https://nostarch.com/gnumake ) which takes things further than the GNU Make manual. A large amount of the content of the book came from a sequence of blog posts on GNU Make by me: https://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-art...

I don't think the FSF is currently selling physical copies of the Make manual.

Ah. I guess I am old and like books. Give them some money instead: https://my.fsf.org/donate/

Re: Makefiles – Best Practices

#110

Post by chubot »

brandmeyer wrote: post #61
chubot wrote: post #25

Earlier quoted context omitted.

FWIW I also read the GNU Make manual, and based some code for automatic deps off a profoundly ugly example it had. Then later people on HN showed me a better/simpler way to do it. https://news.ycombinator.com/item?id=15060149 https://www.gnu.org/software/make/manual/html_node/Automatic... After reading the manual and writing 3 substantial Makefiles from scratch, I still think Make is ugly and, by modern standards, no…

Its not actually all that hard to make well-parallelized makefiles, provided you follow a few basic rules. - Each build step has a unique artifact. - That artifact is visible to Make as a file in the filesystem. - That artifact is named $@ in the rule's recipe. - Every time the recipe is executed, $@ is updated on success. - If the recipe fails, it must return nonzero to Make. - All of the dependencies of the artifac…

That's a nice set of rules, although I don't think they're all easy to follow or verify that 1000 lines of Make is following.

I considered 3 main use cases, and I wrote Makefiles from scratch for all of them. Make works to an extent for each case, but I still have problems.

1. Building mixed Python/C app bundles [1]

2. Building my website [2]. Notably people actually do complain about Jekyll build speed, to the point where they will use a different system like Hugo. So incremental/parallel builds are really useful in this domain!

3. Doing analytics on web log files (e.g. time series from .gz files)

One thing I didn't mention is that they all involve some sort of build parameterization or "metaprogramming". That requirement interacts with the problem of parallel and incremental builds.

For example, for #1, there is logic shared between different bundles. Pattern rules aren't really expressive enough, especially when you have two dimensions. Like (app1, app2, ...) x (debug, release, ASAN, ...)

A pet peeve of mind is having to "make clean" between a debug and a release build, and nearly all usages of Make have that problem, e.g. Python and bash's build system. You could say they are violating your rules because each artifact doesn't have a unique name on the file system (i.e. debug and release versions of the same object file.)

Likewise, Make isn't exactly flexible about how the blog directory structure is laid out. I hit the multiple outputs progblem -- I have Jekyll-style metadata at the front of each post (title, date, tags), so each .md file is split into 2 files. The index.html file depends on all the metadata, but not the data.

All of them have dynamic dependencies too:

1. I generate dependencies using the Python interpreter

2. I add new blog posts without adding Make rules

3. I add new web log files without adding Make rules

Make does handle this to an extent, but there are definitely some latent bugs. I have fixed some of them, but without a good way of testing, I haven't been motivated to fix them all.

I wrote up some more problems in [3], but this is by no means exhaustive. I'm itching to replace all of these makefiles with something that generates Ninja. It's possible I'll hit some unexpected problems, but we'll see.

My usage is maybe a bit out of the ordinary, but I don't see any reason why a single tool shouldn't handle all of these use cases.

[1] Rewriting Python's Build System From Scatch http://www.oilshell.org/blog/2017/05/05.html

[2] http://www.oilshell.org/site.html

[3] Build System Observations http://www.oilshell.org/blog/2017/05/31.html

Post reply on HN