Ninja, a small build system with a focus on speed • Hacker News Board

Re: Ninja, a small build system with a focus on speed

#31

Post by beagle3 » Fri, Apr 27, 2012, 2:44 PM UTC

apenwarr wrote: post #23

beagle3 wrote: post #10

Anyone has an idea of how this compares to apenwarr's implementation of djb's "redo" concept? Compared to make, redo is extremely simple, yet more versatile, more robust - and potentially very efficient. djb only released the spec (not working code). apenwarr implemented it in Python, which means it's a lot slower than it could be (which you'd mostly feel on nop builds).

The short version is that ninja is declarative while redo is imperative. And I'm not sure that battle will be resolved in my lifetime :) Personally, I prefer imperative stuff most of the time, but lots of smart people disagree. ninja config files, as I understand it, are designed to be produced by some other tool, because purely-declarative languages are typically a pain for humans to write by hand. So it's one layer…

> In ninja, that sort of thing would be easier to detect/prevent, and in turn it ought to be easy to implement shared caching, distributed builds, etc in a transparent way.

But apparently, these are not the goals for ninja. The goals for ninja appear to be speed, speed and more speed, especially for a no-op or one-file-change build.

I wonder if anyone converted the build project of a project e.g. the size of chrome to redo and can compare build speeds to the ninja version.

Furthermore, if speed is your major optimization point, it seems the approach taken by http://gittup.org/tup seems impossible to beat, and as a bonus you get perfect dependency information with no additional work (and see http://gittup.org/gittup - they ported quite a few projects to it)

Re: Ninja, a small build system with a focus on speed

#32

Post by beagle3 » Fri, Apr 27, 2012, 2:46 PM UTC

groby_b wrote: post #22

ajross wrote: post #7

Bravo! Granted, it's yet another make replacement the world really doesn't need . But that said, and unlike all the other attemps, this actually seems to be better than make. Almost always, these things are junk (Ant, I'm looking at you) which at best implement a subset of make's features in a "pure" way (and thus look good to people who don't understand make but like Java-or-whatever). This one actually seems to und…

Speaking as somebody who works on a reasonably large project,yes, the world needs it. Null build [^1] with cold cache & make: 40s Null build with cold cache & ninja: 12s Null build with hot cache & make: 20s Null build with hot cache & ninja: Ninja saves me 20 seconds every single time I build something. Let's say I kick off about 30-40 builds a day, that's 10-15 minutes each day . [^1]: I.e. nothing changed

Have you ever considered tup? http://gittup.org/tup seems like it is going to be as fast as ninja, without sacrificing anything. (No project where a null make takes more than 0.1 secs myself, so I can't really tell)

Re: Ninja, a small build system with a focus on speed

#33

Post by cygx » Sat, Apr 28, 2012, 8:24 PM UTC

evmar wrote: post #28

kruhft wrote: post #24

Earlier quoted context omitted.

> multiple outputs for a target I just had to figure out how to do this today with GNU make. It was something like this: %Parser.c %Parser.h %Lexer.c %Lexer.h %.tokens: %.g antlr $ With the object files required by the main project. Worked like a charm and solved the problem of running antlr multiple times for the produced files during a parallel build.

I'm not sure I completely followed your example, but note that the straightforward syntax for multiple outputs from Make does the wrong thing. Here's a discussion of how to make it do what you really want. http://www.gnu.org/software/automake/manual/html_node/Multip...

Look at the last paragraph of the linked site:

For completeness it should be noted that GNU make is able to express rules with multiple output files using pattern rules

The problem with this approach (aside from the fact that it's specific to GNU make) is that you need a shared (non-empty) stem in all output files; however, the following workaround appears to work:

    foo bar baz : % : .dummy/../%
    %/../foo %/../bar %/../baz : quux
    	@sleep 2
    	touch foo bar baz

Re: Ninja, a small build system with a focus on speed

#34

Post by apenwarr » Sun, Apr 29, 2012, 6:37 AM UTC

beagle3 wrote: post #31

apenwarr wrote: post #23

Earlier quoted context omitted.

The short version is that ninja is declarative while redo is imperative. And I'm not sure that battle will be resolved in my lifetime :) Personally, I prefer imperative stuff most of the time, but lots of smart people disagree. ninja config files, as I understand it, are designed to be produced by some other tool, because purely-declarative languages are typically a pain for humans to write by hand. So it's one layer…

> In ninja, that sort of thing would be easier to detect/prevent, and in turn it ought to be easy to implement shared caching, distributed builds, etc in a transparent way. But apparently, these are not the goals for ninja. The goals for ninja appear to be speed, speed and more speed, especially for a no-op or one-file-change build. I wonder if anyone converted the build project of a project e.g. the size of chrome t…

Any project that maintains a purely-declarative dependency tree after the first build should be able to do incremental builds equally fast. That includes ninja, redo, or tup. (Of course, there would be optimization details in the implementation, and ninja is likely to be fastest at present. But the design itself doesn't preclude any of them being fast.)

For full builds, I don't see any reason tup would be particularly fast for speed, in fact. Auto-calculation of dependencies sounds nice, but I personally don't trust it; "perfect" is harder to attain than it sounds. For example, what do you do if one of my build rules retrieves a list of files using wget? redo can handle this, but tup could never automate it "perfectly" (since there are so many possible definitions of perfect), so you will always have weird tradeoffs. I don't really believe in the concept of perfect automated dependencies. Of course, if it works for you, then go for it; that's kind of an edge case.

Re: Ninja, a small build system with a focus on speed

#35

Post by beagle3 » Mon, Apr 30, 2012, 3:31 PM UTC

apenwarr wrote: post #34

beagle3 wrote: post #31

Earlier quoted context omitted.

> In ninja, that sort of thing would be easier to detect/prevent, and in turn it ought to be easy to implement shared caching, distributed builds, etc in a transparent way. But apparently, these are not the goals for ninja. The goals for ninja appear to be speed, speed and more speed, especially for a no-op or one-file-change build. I wonder if anyone converted the build project of a project e.g. the size of chrome t…

Any project that maintains a purely-declarative dependency tree after the first build should be able to do incremental builds equally fast. That includes ninja, redo, or tup. (Of course, there would be optimization details in the implementation, and ninja is likely to be fastest at present. But the design itself doesn't preclude any of them being fast.) For full builds, I don't see any reason tup would be particularl…

> For full builds, I don't see any reason tup would be particularly fast for speed

Indeed.

> "perfect" is harder to attain than it sounds. For example, what do you do if one of my build rules retrieves a list of files using wget?

I wholeheartedly believes that, in that case, you deserve all the suffering your build process calls for :). But seriously, this should be a never-satisfied phony target in any build system.

> I don't really believe in the concept of perfect automated dependencies.

Yes, there is an implicit definition of "perfect" in my writing, and that is: "If any file was consulted in the building of an object previously, and that file has changed, then the dependent object will be rebuilt again".

I don't believe in perfect manual dependencies. So if we are both right, guaranteeing robust builds is not possible :) (which is not an unreasonable conclusion, IMO)

Note that tup makes the implicit dependency on tools explicit (oh, you executed /usr/bin/gcc - that's a dependency. It changed? we need to rebuild). I have never seen any explicit build script do that.

redo can obviously do that -- but have you ever written something like "redo-if-change /usr/bin/gcc"?

> Of course, if it works for you, then go for it; that's kind of an edge case.

I'll be switching from Makefile to redo for my next big non-windows project. I like the idea of tup, but redo's pragmatism is a win for me.

And ... my ideal tool would be a mode for redo which would track execution and file access, and give warnings like "your build depends on /usr/bin/gcc and /usr/include/stdio.h but does not mention it", letting me either make it explicit or ignore it -- but not be ignorant of it.

Re: Ninja, a small build system with a focus on speed

#36

Post by apenwarr » Sat, May 05, 2012, 10:01 PM UTC

beagle3 wrote: post #35

apenwarr wrote: post #34

Earlier quoted context omitted.

Any project that maintains a purely-declarative dependency tree after the first build should be able to do incremental builds equally fast. That includes ninja, redo, or tup. (Of course, there would be optimization details in the implementation, and ninja is likely to be fastest at present. But the design itself doesn't preclude any of them being fast.) For full builds, I don't see any reason tup would be particularl…

> For full builds, I don't see any reason tup would be particularly fast for speed Indeed. > "perfect" is harder to attain than it sounds. For example, what do you do if one of my build rules retrieves a list of files using wget? I wholeheartedly believes that, in that case, you deserve all the suffering your build process calls for :). But seriously, this should be a never-satisfied phony target in any build system.…

I actually have seen build systems that depend on the gcc version (see the 'buildroot' project for example); it's virtually never what you want, because stupid things like installing a tiny bugfix to libc or gcc causes millions of lines of code to rebuild unnecessarily. "Perfect" is not so perfect in that model. For the same reason, a lot of people, myself included, prefer not to include system headers (/usr/include) in their .o file dependencies (and gcc offers a way to decide which you want in your autodeps files).

All that said, I'm not exactly opposed to auto-calculation of dependencies, I just prefer it be optional. Your last paragraph suggests you're okay with such an approach. With that in mind, I think it would be fine to extract out the parts of tup that calculate dependencies, for example, and wrap your .do scripts in that. Then you have the choice.