Polyglot Makefiles

#1

Post by sciencerobot » Fri, May 15, 2020, 4:22 PM UTC

Re: Polyglot Makefiles

#2

Post by mehrdadn » Fri, May 15, 2020, 5:31 PM UTC

I think this might have repercussions? Like if you do "make foo bash bar" then can you predict what that SHELL is used for?

Re: Polyglot Makefiles

#3

Post by brandmeyer » Fri, May 15, 2020, 5:33 PM UTC

That's interesting.

Author probably wants to use `private` for those target-local variables, though.

For example,

    R: .SHELLFLAGS := -e
    R: SHELL := Rscript
    R:
        greeting = "bonjour"
        message(paste0(greeting, ", R!"))

Everything that target `R` depends on will also have SHELL and .SHELLFLAGS over-ridden. If `R` depends on some data generated by another program, it probably wants to be built and executed with the default SHELL (or another shell, perhaps).

    R: private .SHELLFLAGS := -e
    R: private SHELL := Rscript
    R:
        greeting = "bonjour"
        message(paste0(greeting, ", R!"))

Now, `R`'s dependencies will be generated with the makefile's defaults.

Usually I prefer to build up richer stages like this using the system shell anyway, though. Build a target which in turn is executed by the shell normally to traverse the next edge in the graph. But I can see how this mechanism has its uses.

Re: Polyglot Makefiles

#4

Post by epistasis » Fri, May 15, 2020, 5:33 PM UTC

For complicated pipelines that I want to reuse multiple times, I have turned Makefiles into executables by putting this at the top:

    #!/usr/bin/make -f

And then putting them in my $PATH. I run them with arguments like:

    $ process-data.mk INTSV=a.tsv DB=largefile.gz OUTDIR=finished

This makes me feel like I've sold my soul to the devil, and that I'm just living on borrowed time until it all fails and falls apart. It hasn't yet, however...

Re: Polyglot Makefiles

#5

Post by haolez » Fri, May 15, 2020, 5:37 PM UTC

epistasis wrote: post #4

For complicated pipelines that I want to reuse multiple times, I have turned Makefiles into executables by putting this at the top: #!/usr/bin/make -f And then putting them in my $PATH. I run them with arguments like: $ process-data.mk INTSV=a.tsv DB=largefile.gz OUTDIR=finished This makes me feel like I've sold my soul to the devil, and that I'm just living on borrowed time until it all fails and falls apart. It has…

In your scenario, "make processed_data" makes more sense semantically than rules commonly seem in the wild like "make run" or "make deploy"

Re: Polyglot Makefiles

#6

Post by sciencerobot » Fri, May 15, 2020, 5:52 PM UTC

mehrdadn wrote: post #2

I think this might have repercussions? Like if you do "make foo bash bar" then can you predict what that SHELL is used for?

The shell is specific to each target. So doing `make ruby bash python docker` works. It even works in parallel if you do `make -j`.

Edit: I'm the author.

Re: Polyglot Makefiles

#7

Post by sciencerobot » Fri, May 15, 2020, 5:55 PM UTC

brandmeyer wrote: post #3

That's interesting. Author probably wants to use `private` for those target-local variables, though. For example, R: .SHELLFLAGS := -e R: SHELL := Rscript R: greeting = "bonjour" message(paste0(greeting, ", R!")) Everything that target `R` depends on will also have SHELL and .SHELLFLAGS over-ridden. If `R` depends on some data generated by another program, it probably wants to be built and executed with the default S…

I didn't know about private. Thanks for the tip.

Re: Polyglot Makefiles

#8

Post by ainar-g » Fri, May 15, 2020, 5:58 PM UTC

Note that this article (like many, many others) assumes GNU Make. POSIX Make has neither .ONESHELL nor local macros. Neither do most built-in Make implementations in other OSes, like OpenBSD's bmake.

Re: Polyglot Makefiles

#9

Post by mehrdadn » Fri, May 15, 2020, 5:59 PM UTC

sciencerobot wrote: post #6

mehrdadn wrote: post #2

I think this might have repercussions? Like if you do "make foo bash bar" then can you predict what that SHELL is used for?

The shell is specific to each target. So doing `make ruby bash python docker` works. It even works in parallel if you do `make -j`. Edit: I'm the author.

And what happens if there's overlap between the targets?

Re: Polyglot Makefiles

#10

Post by gregwebs » Fri, May 15, 2020, 6:05 PM UTC

Make was designed for building dependencies. I think it is always problematic to use it as a command runner (for example there is no standard way to list out the available commands).

[just](https://github.com/casey/just) is a tool that feels similar to make but is designed explicitly for the purpose of running commands.

I think of this as a simple CLI for your project workflow. You still really want to avoid putting code into a Justfile and put it into scripts. But the Justfile helps provide a slightly nicer UX and automatically invoke dependencies.