Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Language Agnostic, All-Purpose, Incredible, Makefile

blog.mindlessness.life

Post reply on HN
Print view View original

31–40 of 124 posts

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#31

Post by aidenn0 »

gumby wrote: post #28
aidenn0 wrote: post #26

Earlier quoted context omitted.

Consider: foo.c foo.h: foo.in some-tool foo.in If you have some targets that depend on foo.c and other targets that depend on foo.h and you build in parallel, some-tool will run twice, possibly at the same time as something is reading the output, possibly causing problems. [edit] The workaround I've used is something like: foo.c: foo.in some-tool foo.in foo.h: foo.c true Note that the "true" command is important beca…

Ah, now I understand the problem, thanks! Indeed I've stayed away from -j because it's very hard to figure out how recipes may interact when you have a complex Makefile (and with a simple one -j rarely helps much). I like your fix; even without -j, if some-tool is expensive to run you don't want it run redundantly.

I use make (or redo) with -j for parallelizing tasks all the time. It's a really great tool for that.

If you specify your dependencies correctly, then -j will not break anything (other than out-of-memory when parallelizing beyond your memory resources).

Many implementations of redo have an option that will randomize the order of building dependencies; it's a useful way for testing if you have your dependencies fully specified or not.

For non one-off tasks where incremental builds are desired, I've been really liking tup. It will automatically catch many mistakes with dependencies for you up-front, saving much hair-pulling down the road.

[edit]

An example of what I use make/redo parallelization for in one-off scripts is bulk conversion of images or sound files; it's usually 1 file in 1 file out and rarely are the tools already heavily parallel.

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#32

Post by emmelaich »

gumby wrote: post #17
bxparks wrote: post #16

I sometimes use GNU Make to fire off custom code generators, before the files are handed off to other parts of the toolchain which can have their own complicated dependency management. This works quite well. The one annoying problem that I often encounter is that Make does not handle multiple targets (i.e. the code generator generates multiple files, e.g. 'file1.h', 'file1.cpp', 'file2.h', 'file2.cpp', 'test.cpp'). I…

Not sure what you mean by "Make does not handle multiple targets". You can definitely do `make foo bar` and it will run the recipes for both foo and bar. You can also write a recipe with multiple prerequisites (which could be the result of a variable expansion). Curious about the limitation; could be there's a way around it or that I never ran into it.

Not sure what the op has in mind precisely, but Makefiles for modern Fortran with modules is annoying.

Do a search for "makefile fortran modules" or similar and you'll find a bunch of articles, including

https://stackoverflow.com/questions/35234003/how-to-create-a...

I tried helping a friend with some Fortran but eventually gave up due to this issue and GNU Fortran's insistence on using the standard % instead of "." for structure syntax.

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#33

Post by a3n »

m463 wrote: post #23
teddyh wrote: post #2

Make is great, and I wish more people would use it in place of whatever monstrosity is en vogue this week. However, there is one thing which Make absolutely cannot handle, and that is file names with spaces . If you have any risk of encountering these without any possibility of renaming them, you’ll sadly have to give up on using Make; it just won’t work.

Make is useful and the concept is sound. However, the implementation is dated. Just off the top of my head, it could be object oriented (rules could be subclassed), the language could have more sophisticated statements, it could have debugging, etc

It could have non-significant whitespace.

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#34

Post by zwegner »

wwright wrote: post #30
zwegner wrote: post #21

Earlier quoted context omitted.

Note that the author of that paper (a friend of mine) wrote another build system, the dead-simple-but-awesome make.py. I have a mirror/fork of it[0], since it's been unmaintained for a while (but it mostly doesn't need any maintenance). The entire build system is a single Python script that's less than 500 lines of code. Rather than trying to fit complicated rules into Make's arcane syntax, rules are specified with a…

If you haven’t seen Bazel, you should take a look. It’s definitely not as minimalist, but it has a very, very similar model for specifying the build. In me experience, it’s pretty easy to get going, and it makes it pretty hard to screw up any of the important features of the build.

Yeah, I know about Bazel, but only at a high level--I haven't used it.

I generally think the hermetic build concept is a very good one, but IMO Bazel goes about it the wrong way, and is overengineered. Rather than needing custom-built infrastructure for every type of language supported, I'd prefer build systems to use lower level OS facilities for discovering dependencies and controlling nondeterministic behavior. That is, build rules would use something like the rules.py files of make.py, specifying any arbitrary executables to run, but without needing to specify the input dependencies of each rule. Each command run would get instrumented with strace (or the equivalent for non-Linux OSes), and filesystem accesses detected. If a file is opened by a build step, that path would be checked for other build rules. If one exists, and it's out of date, the first build step gets paused while the input file gets built, then resumed. All of this happens recursively for the whole build graph, starting from the first requested build output. Other potentially nondeterministic system calls (timestamps, multi-threading/-processing, network access, etc) would be restricted/controlled in various ways yet to be determined.

That said, I haven't actually built anything like that (or know of anyone else that has). Maybe there's some complicated issues that this couldn't deal with but Bazel could. For example, there might be sources of nondeterminism that don't involve syscalls, like vDSO, I don't know for sure though. Portability between OSes would definitely be an issue. But overall I feel that, barring any major unforeseen issues, something like this could be built in a fairly minimalist fashion; maybe a few thousand lines of Python, possibly a small C module.

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#35

Post by lokedhs »

a3n wrote: post #33
m463 wrote: post #23

Earlier quoted context omitted.

Make is useful and the concept is sound. However, the implementation is dated. Just off the top of my head, it could be object oriented (rules could be subclassed), the language could have more sophisticated statements, it could have debugging, etc

It could have non-significant whitespace.

That would be great, but significant whitespace doesn't seem to stop all the Python users out there.

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#36

Post by lhorie »

wwright wrote: post #30
zwegner wrote: post #21

Earlier quoted context omitted.

Note that the author of that paper (a friend of mine) wrote another build system, the dead-simple-but-awesome make.py. I have a mirror/fork of it[0], since it's been unmaintained for a while (but it mostly doesn't need any maintenance). The entire build system is a single Python script that's less than 500 lines of code. Rather than trying to fit complicated rules into Make's arcane syntax, rules are specified with a…

If you haven’t seen Bazel, you should take a look. It’s definitely not as minimalist, but it has a very, very similar model for specifying the build. In me experience, it’s pretty easy to get going, and it makes it pretty hard to screw up any of the important features of the build.

Unfortunately, Bazel also does not support file names w/ spaces https://github.com/bazelbuild/bazel/issues/374

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#37

Post by peterwwillis »

Makefiles are great entry points for ci/cd pipelines. It's easy to pass arbitrary environment variables at runtime, targets to build, define basic dependencies, and have clear steps to execute that can include some minimal inline shell. And since it's pretty dependency-less, I can run the same make commands locally to test the pipeline as I'd use in a remote CI system.

I often use them as a wrapper for Terraform weirdness, where you may want to call an ADFS-enabled AWS login tool or not, depending on if `aws sts get-caller-identity` returns. Or assume a role before running all targets. Or extract values from a terraform.tfvars.json, to pass to the above two steps. Or bootstrap a remote backend if it doesn't exist. Or remove stale module symlinks. Or properly run init, get, and validate before running a plan or apply. Or document weird -target usage. The end result of just running make prep and make apply with no further knowledge required is exactly the experience I wanted out of Terraform initially.

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#38

Post by goblin89 »

Unlike a target, .PHONY can be populated incrementally. For example, this:

    .PHONY: serve live-reload

    serve: init deps compile db-setup db-migrate
        rails server

    live-reload: yarn
        ./bin/webpack-dev-server --host 127.0.0.1
can become this, making things a tiny bit easier to maintain when there are many targets:

    .PHONY: serve
    serve: init deps compile db-setup db-migrate
        rails server

    .PHONY: live-reload
    live-reload: yarn
        ./bin/webpack-dev-server --host 127.0.0.1
Recently I joined an environment that uses Makefiles as the facade in front of pretty much everything, from git submodule update shortcuts to building code and running local development servers.

Surprising myself, I’ve quickly grown to appreciate working Makefiles. That said, since the syntax somewhat encourages terseness, when I need to fix a non-trivial target it tends to look like black magic—nothing reading a few man pages can’t fix, but it takes extra time.

It’s not my first choice overall, I prefer to leave out the extra layer and document direct command-line calls in a README. If a commonly used tool changes its call in a new version, with README it’s a documentation issue, but with Makefile it’s broken software.

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#39

Post by upc031901 »

teddyh wrote: post #2

Make is great, and I wish more people would use it in place of whatever monstrosity is en vogue this week. However, there is one thing which Make absolutely cannot handle, and that is file names with spaces . If you have any risk of encountering these without any possibility of renaming them, you’ll sadly have to give up on using Make; it just won’t work.

Spaces in filenames break most of make's builtin functions such as $(sort), and break the $?, $^ and $+ automatic variables. But they're OK in target names as long as you escape the spaces with backslashes. In some cases you can also use them in source file names -- you have to hard code the names in the build rules since $^ won't work (but for targets built from a single source file, $This applies to GNU make; not sure about AT&T make.

Re: The Language Agnostic, All-Purpose, Incredible, Makefile

#40

Post by m463 »

lokedhs wrote: post #35
a3n wrote: post #33

Earlier quoted context omitted.

It could have non-significant whitespace.

That would be great, but significant whitespace doesn't seem to stop all the Python users out there.

I will say I like how python uses whitespace syntactically. The whitespace is consistent, makes programs readable and much more compact.

make seems to be less consistent. I think it distinguishes between a space and a tab, which isn't clear without editor help.

Post reply on HN