Why Use Make • Hacker News Board

Re: Why Use Make

#221

Post by cdavid » Mon, Feb 25, 2013, 5:39 PM UTC

cmccabe wrote: post #215

cdavid wrote: post #202

Earlier quoted context omitted.

Using a macro language was already bad in the 90ies (autoconf/automake using m4), but using one in 2000 is just tragic. It is also impossible to debug, partly because of the abomination the cmake language is: even understanding where a variable is defined is hard, and because the language is so unexpressive, the Find*.cmake modules are often in the 1000s lines count. For all its suckiness, I take autoconf/automake ov…

I'm sorry, but your comment does not make much sense to me. autotools is mostly written in m4, with some shell snippets. Those are macro languages. If you don't like macro languages, logically you should not like autotools. CMake is not "impossible to debug." In fact, it is a lot easier to debug than autotools-- partly because you end up writing so much less code. Also you don't have the three levels of "generated fi…

My point was that autotools had the excuse of being written in the early 90ies, cmake doesn't.

Your experience with debugging autotools vs cmake does not match mine: cmake is not an improvement over autotools (if only because at least with autotools, there is some decent doc out there and google knows a lot about autoconf insanity). It took me hours to debug trivial issues with cmake, because you can't easily trace where variables are defined.

Re: Why Use Make

#222

Post by rachelbythebay » Mon, Feb 25, 2013, 6:06 PM UTC

lucian1900 wrote: post #220

rachelbythebay wrote: post #74

Earlier quoted context omitted.

Assuming you're building your own code and are willing to arrange things in a compatible manner, then I have a solution to offer. It figures out the dependencies by reading the source code . You can create objects and binaries just by asking it to build a certain target. I recorded an example of installing and using it here: http://rachelbythebay.com/jvt/view?bb_install You can get a copy for experimentation here: ht…

That is very cool, and it's something that will only get better if the Apple-proposed modules get widespread.

Thank you! I'm not sure what you mean by "Apple-proposed modules", though. Please let me know more and I'll see what I can do.

Re: Why Use Make

#223

Post by mshal » Mon, Feb 25, 2013, 6:15 PM UTC

quotemstr wrote: post #133

payload wrote: post #88

Earlier quoted context omitted.

Thanks for the links! I will definitely use tup for some next project. I try from time to time different make-like systems but I always come back to GNU Make. I also don't fear the GNU Make Reference. But tup looks, again, quite promising. It even has those little context sensitive one-special-char one-letter variables :) But they do something better by design, I see http://gittup.org/tup/make_vs_tup.html

> This page compares make to tup. This page is a little biased because tup is so fast. How fast? This one time a beam of light was flying through the vacuum of space at the speed of light and then tup went by and was like "Yo beam of light, you need a lift?" cuz tup was going so fast it thought the beam of light had a flat tire and was stuck. True story. Anyway, feel free to run your own comparisons if you don't beli…

You might find my paper more informative and less unprofessional: http://gittup.org/tup/build_system_rules_and_algorithms.pdf

Many projects are bottlenecked on the build system. You can benchmark this by timing a null build (running 'time make' after building everything). Some examples from my machine are the Linux kernel (28 seconds), and Firefox (1m 23 seconds). Some of this time is from unnecessarily recompiling things, but that is a separate issue from the inherent lack of scalability in make.

Suppose I want to change a single C/C++ file in one of these projects - the total turnaround time from when I type 'make' to when the build finishes can be described as:

T(total) = T(build system) + T(sub-processes)

Ideally T(total) would be zero, meaning we get an instant response from when we change the file to when we can test the result. Here, T(build system) is the null build time, and T(sub-processes) is the time it takes to run the compiler and such. Using the Linux kernel as an example again, compiling fs/ext3/balloc.c takes 0.478 seconds. In comparison to the null build of 28 seconds, there are significant gains to be had by optimizing T(build system).

Amdahl's Law is a little tricky to apply since tup is not parallelizing T(build system), but rather changing it from a linear-time algorithm to a logarithmic-time algorithm. So you can set P easily based on the relative values of T(build system) and T(sub-processes), but S is not a simple "count-the-cores" metric. The speedup is effectively N/log(N), where N is the number of files. This is much better than simple parallelization - T(build system) for tup with these projects is only about 4ms. The total turnaround time for the balloc.c file in the Linux kernel is 1.1 seconds (which includes compilation and all the linking steps afterward), in comparison to make's total turnaround time of 29.5 seconds.

Re: Why Use Make

#224

Post by rachelbythebay » Mon, Feb 25, 2013, 6:21 PM UTC

ysangkok wrote: post #182

rachelbythebay wrote: post #74

Earlier quoted context omitted.

Assuming you're building your own code and are willing to arrange things in a compatible manner, then I have a solution to offer. It figures out the dependencies by reading the source code . You can create objects and binaries just by asking it to build a certain target. I recorded an example of installing and using it here: http://rachelbythebay.com/jvt/view?bb_install You can get a copy for experimentation here: ht…

Is the source available?

If you mean the source to my replay stuff, well, it seems you have already established a way to snag that. The only original part of that was my wrapper to the terminal to fetch byte streams and replay them while honoring the timing data.

Regarding the build tool source, I haven't decided what to do about that just yet. It could be particularly valuable in a corporate environment.

Re: Why Use Make

#225

Post by mshal » Mon, Feb 25, 2013, 6:26 PM UTC

Peaker wrote: post #84

chubot wrote: post #81

Earlier quoted context omitted.

What are the shortcomings of tup? That you have to specify dependencies statically?

Yeah, though it is not that horrible since tup allows you to over-specify dependencies with very minor ill-effects (less parallelism, but no over-building). My list of short-comings besides that are: * No "run" command on Windows, means if you want portable tup you're stuck with its primitive scripting language (or reverting to Make-style hacks like ugly multi-phase build that generates the Tupfiles) * Has an arbitra…

Have you tried Rendaw's lua branch at all? I'm curious if that would remove your need for the 'run' command. At some point that will be merged to the mainline. Here's his tree: https://github.com/Rendaw/tup

Is the "Line too long" error an error message from tup, or from the shell when it tries to fork a process? If it's the latter I'm not sure there is an easy fix. If it's the former maybe I was just lazy when implementing something :)

You can run sub-processes in a chroot by specifying a flag (search the man page for 'run inside a chroot'). This will prevent the fuse paths from leaking to sub-processes, but unfortunately it requires the tup executable to be suid root. (If it didn't need suid, this would be the default).

Re: Why Use Make

#226

Post by kragen » Mon, Feb 25, 2013, 6:27 PM UTC

rachelbythebay wrote: post #179

_phred wrote: post #171

Earlier quoted context omitted.

Whoa, your terminal playback thing is pretty neat. Did you use GNU Screen to record the session?

Thank you! I used script. Certain implementations have an option to emit timing and byte count data on stderr. Then it's just a matter of saving it and building something to honor those delays during playback.

That's awesome! I had no idea about `script -t`; I'd written my own a few years back. Thank you!

Re: Why Use Make

#227

Post by kragen » Mon, Feb 25, 2013, 6:27 PM UTC

rachelbythebay wrote: post #179

_phred wrote: post #171

Earlier quoted context omitted.

Whoa, your terminal playback thing is pretty neat. Did you use GNU Screen to record the session?

Thank you! I used script. Certain implementations have an option to emit timing and byte count data on stderr. Then it's just a matter of saving it and building something to honor those delays during playback.

That's awesome! I had no idea about `script -t`; I'd written my own a few years back. Thank you!

Re: Why Use Make

#228

Post by lucian1900 » Mon, Feb 25, 2013, 6:31 PM UTC

rachelbythebay wrote: post #222

lucian1900 wrote: post #220

Earlier quoted context omitted.

That is very cool, and it's something that will only get better if the Apple-proposed modules get widespread.

Thank you! I'm not sure what you mean by "Apple-proposed modules", though. Please let me know more and I'll see what I can do.

http://llvm.org/devmtg/2012-11/Gregor-Modules.pdf

Re: Why Use Make

#229

Post by kragen » Mon, Feb 25, 2013, 6:33 PM UTC

stephth wrote: post #31

largesse wrote: post #19

Created in 1977, Make has its quirks. But whether you prefer GNU Make or a more recent alternative, consider the benefits of capturing your workflow in a machine-readable format. Like a programming language?

Exactly what I was thinking. Use a scripting language you're proficient at. This article doesn't make Make look worth the trouble if you don't have a more specific reason to use it. Too much complexity, and quirks like you’ll need to delete the previously-downloaded zip file before running make can easily be avoided with a scripting language. These days any relatively convoluted task that I may need to do more than o…

If you're using a series of shell commands, each of which takes seconds to hours to run, you will be happy with make's ability to (1) run shell commands easily and (2) not run them again when they don't need to be run. You could write a dependency tracking thingy in any programming language, but at that point you might as well be using Rake or some other make replacement.

Re: Why Use Make

#230

Post by SomeCallMeTim » Mon, Feb 25, 2013, 7:25 PM UTC

ycombobreaker wrote: post #195

SomeCallMeTim wrote: post #189

Earlier quoted context omitted.

>So where have you seen it suck the most? When people (myself included) start taking advantage of the fact that Make is Turing-complete and writing arbitrary "programs" in their Makefiles. It typically starts simple; you want to do something like build ALL the files in a folder, so you use a wildcard. Then you want to add dependency checking, so you use the wildcard to convert between .o to .d, keeping the same folde…

> some of my Make variables are referenced with $(VAR), and some with $$(VAR), depending on whether I want them to grab the CURRENT version of the variable or the calculated version. Hah, my latest Makefile work has been a set of functions which generate Make-syntax output, which then gets $(eval)ed. I hear you on the debugging nightmare that this can be: does a given variable get resolved when the function is first…

I also use "printf debugging"; I have to.

The worst problem I had, though, was REALLY annoying; I was getting an inscrutable error in the middle of a function, and I could delete large parts of the code to get the error to go away, but putting ANY of the code back brought the error back -- it didn't matter which parts I put back.

It turned out that git had changed LF to CRLF in the file, and some end-of-line character was screwing up the spacing. Tweaking .gitattributes and fixing the files made everything work.

I SO hate significant white-space. I never really forgave Python for that "feature" either. But I could totally get behind Lua for the logic. :)

Actually, if it were my job, I would use LuaJIT to write a make replacement; the dependencies could all be specified in tables or extended strings, and any more complicated logic could be explicitly outside of the "rules".

>but the problem is, the _rest_ of the build, which more people interact with on a daily basis, gets more complex when that happens

I think a good design would NOT have that problem. You could have it say "these files get built by default rules" separately from "these rules trigger this bit of Lua code, which can spit out warnings, add dependencies dynamically (oh wouldn't THAT be nice!), or do this other bit of complicated build processing that doesn't fit well into the rule-based system".

If you're doing it in Makefiles, then yes, you could make everything more complicated that way. But I think a fresh design could really do a good job in killing make. I'm just so busy with other things right now, though...

Another reason I would STRONGLY choose Lua over any other scripting system is that the entire tool can embed Lua trivially, while Python or Ruby or Perl would each bring an entire ecosystem with it. You can have a dozen different Lua installs on your system without requiring a separate infrastructure for managing Lua installs.