Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A Tutorial on Portable Makefiles

nullprogram.com

Post reply on HN
Print view View original

91–100 of 114 posts

Re: A Tutorial on Portable Makefiles

#91

Post by pjmlp »

bonzini wrote: post #90
benesch wrote: post #81

Earlier quoted context omitted.

I would love for this to be true, but I'm afraid I don't have any evidence that it is. What are the better options you'd recommend? More specifically, Google's Bazel [0] is the most serious attempt I know of to replace CMake/autotools, but it only just began to support Windows [1] and requires the JDK. Say what you will about CMake, but it's a quick-and-easy MSI to install on Windows and has supported Windows for pro…

Meson ( https://www.mesonbuild.com/ ) seems to have a lot of momentum; GNOME has just adopted it. It is designed around a non-Turing complete, object-oriented DSL, plus extension modules written in Python.

Beyond being adopted by GNOME, I have hardly heard of it being used in another context.

Re: A Tutorial on Portable Makefiles

#92

Post by c3d »

This article barely addresses what really causes trouble in practice, namely non-portable tools. Sed for example has different switches on macos and linux. MinGW is another world.

Also check out https://github.com/c3d/build for a way to deal with several of the issues the author addresses (but not posix portability)

Re: A Tutorial on Portable Makefiles

#93

Post by signa11 »

pjmlp wrote: post #91
bonzini wrote: post #90

Earlier quoted context omitted.

Meson ( https://www.mesonbuild.com/ ) seems to have a lot of momentum; GNOME has just adopted it. It is designed around a non-Turing complete, object-oriented DSL, plus extension modules written in Python.

Beyond being adopted by GNOME, I have hardly heard of it being used in another context.

> ... I have hardly heard of it being used in another context...

dpdk seems to be moving in that direction i.e. using meson.

Re: A Tutorial on Portable Makefiles

#94

Post by majewsky »

zbuf wrote: post #32
pavlov wrote: post #15

Earlier quoted context omitted.

Yes, base Make is almost useless for anything larger than a utility.

Not quite; I think '.c.o' is the equivalent in the original Make syntax?

That's nowhere near equivalent. For example, it cannot replace the following things:

  pkg/test/migrations/%.sql: pkg/db/migrations/%.sql

  build/%.cover.out: prepare-check FORCE
(In the second one, $* is then used in the build rule, which is why % does not need to show up in the prerequisites list.)

Examples from https://github.com/sapcc/limes/blob/62e07b430e2019a6c1891443...

Re: A Tutorial on Portable Makefiles

#95

Post by vog »

beagle3 wrote: post #88
tinus_hn wrote: post #41

Earlier quoted context omitted.

Make has no memory so it can't remember things. It simply compares the dates of files. If a dependency is newer than a target the target is rebuilt. If you want to keep some kind of memory you have to build and keep track of it yourself. But the problem you point at is simply poor design. It is not a normal occurrence for system header files to move around like you state. If they do, a full rebuild is indeed required…

If an apt-get upgrade fixed an issue in a system header or a library, but the date of the fix predates the last build (quite common; last build from yesterday, fix from two days ago but downloaded today) then make will do nothing (or any subset of the right things but not all) and make clean; make will do the right thing. Relying on time stamps is a design decision that was good for its time, but it is no longer robu…

> Relying on time stamps is a design decision that was good for its time, but it is no longer robust

I agree with the sentiment, but a small nitpick:

Relying on time stamps for older/newer comparisons is not robust.

Using time stamps (and perhaps file size) for equality checks is quite robust. And the combination with cryptographic hashes is even better (if a file is recreated but has the same contents afterwards, timestamp checks would trigger an unneeded rebuild, while a crypto hash check would recognize that there's nothing to rebuild).

Re: A Tutorial on Portable Makefiles

#96

Post by susam »

We invoke shell commands in a Makefile, and if we are concerned about POSIX conformance in the Makefile syntax, we need to be equally concerned about POSIX conformance in the shell commands and the shell scripts we invoke in Makefile.

While I have not found a foolproof way to test for and prove POSIX conformance in shell scripts, I usually go through the POSIX.1-2001 documents to make sure I am limiting my code to features specified in POSIX. I test the scripts with bash, ksh, and zsh on Debian and Mac. Then I also test the scripts with dash, posh and yash on Debian. See https://github.com/susam/vimer/blob/master/Makefile for an example.

Here are some resources:

* POSIX.1-2001 (2004 edition home): http://pubs.opengroup.org/onlinepubs/009695399/

* POSIX.1-2001 (Special Built-In Utilities): http://pubs.opengroup.org/onlinepubs/009695399/idx/sbi.html

* POSIX.1-2001 (Utilities): http://pubs.opengroup.org/onlinepubs/009695399/idx/utilities...

* POSIX.1-2008 (2016 edition home): http://pubs.opengroup.org/onlinepubs/9699919799/

* POSIX.1-2008 (Special Built-In Utilities): http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3...

* POSIX.1-2008 (Utilities): http://pubs.opengroup.org/onlinepubs/9699919799/idx/utilitie...

The editions mentioned in parentheses are the editions available at the mentioned URLs at the time of posting this comment.

Here is a list of the commands specified in POSIX:

Special Built-In Utilities: break, colon, continue, dot, eval, exec, exit, export, readonly, return, set, shift, times, trap, unset

Utilities: admin, alias, ar, asa, at, awk, basename, batch, bc, bg, c99, cal, cat, cd, cflow, chgrp, chmod, chown, cksum, cmp, comm, command, compress, cp, crontab, csplit, ctags, cut, cxref, date, dd, delta, df, diff, dirname, du, echo, ed, env, ex, expand, expr, false, fc, fg, file, find, fold, fort77, fuser, gencat, get, getconf, getopts, grep, hash, head, iconv, id, ipcrm, ipcs, jobs, join, kill, lex, link, ln, locale, localedef, logger, logname, lp, ls, m4, mailx, make, man, mesg, mkdir, mkfifo, more, mv, newgrp, nice, nl, nm, nohup, od, paste, patch, pathchk, pax, pr, printf, prs, ps, pwd, qalter, qdel, qhold, qmove, qmsg, qrerun, qrls, qselect, qsig, qstat, qsub, read, renice, rm, rmdel, rmdir, sact, sccs, sed, sh, sleep, sort, split, strings, strip, stty, tabs, tail, talk, tee, test, time, touch, tput, tr, true, tsort, tty, type, ulimit, umask, unalias, uname, uncompress, unexpand, unget, uniq, unlink, uucp, uudecode, uuencode, uustat, uux, val, vi, wait, wc, what, who, write, xargs, yacc, zcat

Re: A Tutorial on Portable Makefiles

#97

Post by pm215 »

beagle3 wrote: post #88
tinus_hn wrote: post #41

Earlier quoted context omitted.

Make has no memory so it can't remember things. It simply compares the dates of files. If a dependency is newer than a target the target is rebuilt. If you want to keep some kind of memory you have to build and keep track of it yourself. But the problem you point at is simply poor design. It is not a normal occurrence for system header files to move around like you state. If they do, a full rebuild is indeed required…

If an apt-get upgrade fixed an issue in a system header or a library, but the date of the fix predates the last build (quite common; last build from yesterday, fix from two days ago but downloaded today) then make will do nothing (or any subset of the right things but not all) and make clean; make will do the right thing. Relying on time stamps is a design decision that was good for its time, but it is no longer robu…

Typically if a system header has changed or been added due to upgrading a library package you'll need to rerun any configure script anyway (since it very likely checks for header features and those decisions will change). So unless your build system magically makes configure depend on every header file used in every configure test it runs, you'll need to redo a clean build anyway, pretty much.

Make has a whole pile of issues, but this one really isn't an aggravation in practice, I find.

Re: A Tutorial on Portable Makefiles

#98

Post by JdeBP »

boris wrote: post #86
erlehmann_ wrote: post #2

An issue I have with make is that it can not handle non-existence dependencies. DJB noted this in 2003 [1]. To quote myself on this [2]: > Especially when using C or C++, often target files depend on nonexistent files as well, meaning that a target file should be rebuilt when a previosly nonexistent file is created: If the preprocessor includes /usr/include/stdio.h because it could not find /usr/local/include/stdio.h…

Could you summarize how you handle this in redo? Also what about the case where a header file does exist but is out-of-date and because of that triggers an error (e.g., version compatibility check with #error) -- how do you handle that?

I, for one, handle it with a tool that mimics the compiler's preprocessing phase and emits both redo-ifchange information for all of the headers that are used, and redo-ifcreate information for all of the non-existent headers that are looked for during the process.

    JdeBP %cat test.cpp
    #include 
    void f() {}
    JdeBP %/package/prog/cc/command/cpp test.cpp --iapplication . --icompiler-high /usr/local/lib/gcc5/include/c++ --icompiler-low /usr/local/lib/gcc5/include/c++/x86_64-portbld-freebsd10.3 --iplatform /usr/local/include --iplatform /usr/include  -MD -MF /dev/stderr 2>&1 > /dev/null|fgrep redo
    redo-ifcreate ./cstddef ./bits/c++config.h /usr/local/lib/gcc5/include/c++/bits/c++config.h /usr/local/include/bits/c++config.h /usr/include/bits/c++config.h ./bits/os_defines.h /usr/local/lib/gcc5/include/c++/bits/os_defines.h /usr/local/include/bits/os_defines.h /usr/include/bits/os_defines.h ./bits/cpu_defines.h /usr/local/lib/gcc5/include/c++/bits/cpu_defines.h /usr/local/include/bits/cpu_defines.h /usr/include/bits/cpu_defines.h ./stddef.h /usr/local/lib/gcc5/include/c++/stddef.h /usr/local/include/stddef.h ./sys/cdefs.h /usr/local/lib/gcc5/include/c++/sys/cdefs.h /usr/local/include/sys/cdefs.h ./sys/_null.h /usr/local/lib/gcc5/include/c++/sys/_null.h /usr/local/include/sys/_null.h ./sys/_types.h /usr/local/lib/gcc5/include/c++/sys/_types.h /usr/local/include/sys/_types.h ./machine/_types.h /usr/local/lib/gcc5/include/c++/machine/_types.h /usr/local/include/machine/_types.h ./x86/_types.h /usr/local/lib/gcc5/include/c++/x86/_types.h /usr/local/include/x86/_types.h
    redo-ifchange /usr/local/lib/gcc5/include/c++/cstddef /usr/local/lib/gcc5/include/c++/x86_64-portbld-freebsd10.3/bits/c++config.h /usr/local/lib/gcc5/include/c++/x86_64-portbld-freebsd10.3/bits/os_defines.h /usr/local/lib/gcc5/include/c++/x86_64-portbld-freebsd10.3/bits/cpu_defines.h /usr/include/stddef.h /usr/include/sys/cdefs.h /usr/include/sys/_null.h /usr/include/sys/_types.h /usr/include/machine/_types.h /usr/include/x86/_types.h
    JdeBP %/package/prog/cc/command/cpp test.cpp --iapplication . --icompiler-high /usr/local/lib/gcc5/include/c++ --icompiler-low /usr/local/lib/gcc5/include/c++/x86_64-portbld-freebsd10.3 --iplatform /usr/local/include --iplatform /usr/include  -MMD -MF /dev/stderr 2>&1 > /dev/null|fgrep redo
    redo-ifcreate ./cstddef ./bits/c++config.h ./bits/os_defines.h ./bits/cpu_defines.h ./stddef.h ./sys/cdefs.h ./sys/_null.h ./sys/_types.h ./machine/_types.h ./x86/_types.h
    redo-ifchange
    JdeBP %
I also have a wrapper that takes arguments in the forms that one would invoke g++ -E and clang++ -E, tries to works out all of the platform and compiler include paths, and invokes this tool with them.

It's then a simple matter of invoking these redo-ifchange and redo-ifcreate commands from within the redo script that is invoking the compiler.

You can see this plumbed into redo in a real system in the source archives for the nosh toolset and djbwares.

* http://jdebp.eu./FGA/introduction-to-redo.html#CompilerDefic...

* https://news.ycombinator.com/item?id=15044438

Re: A Tutorial on Portable Makefiles

#99

Post by JdeBP »

Matheus28 wrote: post #25
erlehmann_ wrote: post #9

Earlier quoted context omitted.

Please elaborate: What do you find amazing about scons? Also, how does scons handle non-existence dependencies? What would be a scons dependency graph for this C code? #include main() { printf("hello, world\n"); return 0; } You can see a dependency graph I generated with redo here: http://news.dieweltistgarnichtso.net/posts/redo-gcc-automati...

I love that I get to use python to write the dependency graph, it allows for some interesting stuff. Other than that, it's mostly the ease of use. This is enough to compile a C++ project (that has all its .c and .cpp files in the same directory as the SConstruct file), and it'll pick up on all dependencies correctly: Program(target = 'a.out', source = Glob('*.c') + Glob('*.cpp')) I also know for a fact that it's able…

From what you are showing us, the answer to How does scons handle non-existence dependencies? is that it does not handle them at all.

Go and look at M. Moskopp's graph. It has a lot of dependencies for non-existent files that the compiler would have used in preference to the ones that it actually used, had they existed.

Re: A Tutorial on Portable Makefiles

#100

Post by erlehmann_ »

JdeBP wrote: post #98
boris wrote: post #86

Earlier quoted context omitted.

Could you summarize how you handle this in redo? Also what about the case where a header file does exist but is out-of-date and because of that triggers an error (e.g., version compatibility check with #error) -- how do you handle that?

I, for one, handle it with a tool that mimics the compiler's preprocessing phase and emits both redo-ifchange information for all of the headers that are used, and redo-ifcreate information for all of the non-existent headers that are looked for during the process. JdeBP %cat test.cpp #include void f() {} JdeBP %/package/prog/cc/command/cpp test.cpp --iapplication . --icompiler-high /usr/local/lib/gcc5/include/c++ --…

I use strace(1) look for stat(2) syscalls that fail with ENOENT. An advantage of this approach is that I do not have to imitate the C preprocessor, so parser differentials can never happen. The following default.o.do file from my blog post [1] handles the case:

  #!/bin/sh
  redo-ifchange $2.c
  strace -e stat,stat64,fstat,fstat64,lstat,lstat64 -f 2>&1 >/dev/null\
   gcc $2.c -o $3 -MD -MF $2.deps\
   |grep '1 ENOENT'\
   |grep '\.h'\
   |cut -d'"' -f2 2>/dev/null\
   >$2.deps_ne
  
  read d
This approach is also used for building Liberation Circuit if strace is installed [2].

I think the compiler should output the necessary information. To quote Jonathan de Boyne Pollard [3]:

> As noted earlier, no C or C++ compiler currently generates any redo-ifcreate dependency information, only the redo-ifchange dependency information. This is a deficiency of the compilers rather than a deficiency of redo, though. That the introduction of a new higher-precedence header earlier on the include path will affect recompilation is a fact that almost all C/C++ build systems fail to account for.

> I have written, but not yet released, a C++ tool that is capable of generating both redo-ifchange information for included files and redo-ifcreate information for the places where included files were searched for but didn't exist, and thus where adding new (different) included files would change the output.

JdeBP, could you please release your tool under a free software license? I suspect it has fewer errors than the similar CMake approach [4].

[1] http://news.dieweltistgarnichtso.net/posts/redo-gcc-automati...

[2] https://github.com/linleyh/liberation-circuit/blob/master/sr...

[3] http://jdebp.eu./FGA/introduction-to-redo.html#CompilerDefic...

[4] https://github.com/Kitware/CMake/blob/master/Source/cmDepend...

Post reply on HN