Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster Than Ninja

build2.org

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Faster Than Ninja

#31

Post by Orphis »

evmar wrote: post #28
Alecazam wrote: post #26

Earlier quoted context omitted.

I feel like make/ninja suffer from the same output modstamp > input modstamps to rebuild. Really a build system should track modstmaps on all inputs (apps included), and the rebuild on not-equal. Would be nice to see build2 go this route.

I did some exploration of this idea in a followup build system! See https://neugierig.org/software/blog/2022/03/n2.html . (It's not really production-ready.)

Why not just embrace Content Based Addressing and the Bazel action cache? Not necessarily adopt the RBE protos, but you could have on-disk cache that are similar or even identical.

Nowadays, I think a build tool that doesn't natively support distributed caching (and possibly remote execution) is a weird choice. And I don't think that spawning layers of processes allows for good parallelization as you don't know if an action is going to be network bound or compute bound and the job slot is then spent. So you either oversubscribe or undersubscribe.

Re: Faster Than Ninja

#32

Post by vlovich123 »

boris wrote: post #16
evmar wrote: post #4

[Ninja author here] Nice post, cool to see the deep dive! I also appreciate the details on how they produced their numbers. As they observe, Ninja gets to be fast mostly by cheating: it avoids a lot of work by saying many things are just out of scope for Ninja to do, and that means it is a useful a target to race against. (Funny thing: when I wrote Ninja I was misremembering how fast an earlier build system was so I…

[build2 author here] Thanks for the feedback! Some additional details: > The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why. Fair enough. The point I was making is that if you want to compete with Ninja, you cannot leave any potential performance gains on the table. > The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible r…

> > They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2. I measured this, it costs 70ms or ~2% of the overall time.

Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?

Re: Faster Than Ninja

#33

Post by yaris »

evmar wrote: post #28
Alecazam wrote: post #26

Earlier quoted context omitted.

I feel like make/ninja suffer from the same output modstamp > input modstamps to rebuild. Really a build system should track modstmaps on all inputs (apps included), and the rebuild on not-equal. Would be nice to see build2 go this route.

I did some exploration of this idea in a followup build system! See https://neugierig.org/software/blog/2022/03/n2.html . (It's not really production-ready.)

Reading about manifests immediately reminded be of Clearcase/Clearmake. My memory may be not correct after all the years passed, but configuration records for derived objects contained the same information as proposed manifests + some more metadata about source (versions, at least). If I remember correctly SCons allows to implement a custom "checker" (or whatever it is called there) to compare files based on any criteria that are useful, not just timestamps. But this is about the only visible advantage of SCons over Make, at least for our case.

Re: Faster Than Ninja

#34

Post by boris »

vlovich123 wrote: post #32
boris wrote: post #16

Earlier quoted context omitted.

[build2 author here] Thanks for the feedback! Some additional details: > The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why. Fair enough. The point I was making is that if you want to compete with Ninja, you cannot leave any potential performance gains on the table. > The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible r…

> > They also mention doing other things like invoking the compiler to get version information. This seems like it would dwarf any performance gain from number 2. I measured this, it costs 70ms or ~2% of the overall time. Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?

> Couldn’t you amortize this to 0 by just caching the result and only changing it if the binary timestamp changed?

Yes, that would be nice, but the tricky question is can any of this information change without the compiler binary mtime changing? First off, GCC's gcc/g++ binaries are drivers and are not what does the actual compilation, it's private cc1/cc1plus binaries that do the job. Can one of these change but not the driver? I think it's plausible (some package manager optimization where the file is not touched if it hasn't changed). So at a minimum we would need to discover where those are located (probably by invoking gcc/g++) and checking them as well. Could there be something else? Who knows. We value speed very much but we value correctness even more.

I think a more fruitful direction to explore is to improve GCC itself to dump all this information in a single invocation and in a machine-readable format (JSON). I think if we go from 70ms to 14ms (and perhaps even lower because this special GCC mode could conceivably do things faster than how we do it now), it would be good enough.

Re: Faster Than Ninja

#35

Post by boris »

zamalek wrote: post #11

> Let's see if we can go even faster. Next, we disable compression in the file cache. We will discuss the file cache in more detail a bit later but for now let's just say that by disabling compression we trade temporary disk space usage for speed: Something is wrong here. Which compression algorithm is being used here and how much has it been tuned? A core hypothesis of the likes of zram is that disk access is so slo…

It is lz4. While all that you say about disk being slow makes sense, measurements show that in this specific situation disabling compression improves performance by a little. My hand-wavy explanation is that freeing CPU to doing compilation rather than [de]compression is a better use of it, with all the uncompressed disk writes probably ending up in the system's file cache.

Re: Faster Than Ninja

#36

Post by majoe »

evmar wrote: post #28
Alecazam wrote: post #26

Earlier quoted context omitted.

I feel like make/ninja suffer from the same output modstamp > input modstamps to rebuild. Really a build system should track modstmaps on all inputs (apps included), and the rebuild on not-equal. Would be nice to see build2 go this route.

I did some exploration of this idea in a followup build system! See https://neugierig.org/software/blog/2022/03/n2.html . (It's not really production-ready.)

I stumbled upon the very experimental nix-ninja [0] recently.

They seem to create a dynamic nix derivation per compilation unit, which would be very similar to what you describe as manifests in your post as it also creates a hash of all inputs.

Would be interesting to here your opinion on that approach

[0]: https://github.com/pdtpartners/nix-ninja

Re: Faster Than Ninja

#37

Post by menaerus »

boris wrote: post #16
evmar wrote: post #4

[Ninja author here] Nice post, cool to see the deep dive! I also appreciate the details on how they produced their numbers. As they observe, Ninja gets to be fast mostly by cheating: it avoids a lot of work by saying many things are just out of scope for Ninja to do, and that means it is a useful a target to race against. (Funny thing: when I wrote Ninja I was misremembering how fast an earlier build system was so I…

[build2 author here] Thanks for the feedback! Some additional details: > The first one is a criticism of CMake, not Ninja (?), so I don't think it can be why. Fair enough. The point I was making is that if you want to compete with Ninja, you cannot leave any potential performance gains on the table. > The second reason given is doing some work like header dependencies in multiple threads. This is the most plausible r…

> It fits into the system's file cache unless there is memory pressure, like one would expect from having 24 C++ compiler jobs running in parallel. We actually measured this in isolation (with more detailed results in the linked article) and it has a measurable effect.

300 TUs is not much. If they build in 3 seconds then they are trivial (small). If the machine is 24-thread (I assume some sort of heterogeneous 12-core), how little RAM does the machine have for the kernel to start evicting page cache during the build?

Re: Faster Than Ninja

#38

Post by bluGill »

vient wrote: post #29
bluGill wrote: post #14

Cmake spend 15 seconds to generate this project? I find it unlikely build2 is doing the same work as cmake is. Now I will grant cmake is single threaded and slow, so there is a lot of room to do the things it does better (the language sucks, and is part of what forces single threaded). It also wouldn't surprise me if cmake is doing things that are not really needed (odds are the default compiler works - most of the t…

Not only is CMake single threaded but the generation step time scales almost quadratically* with the number of targets in some cases. Probably not what happens here but this is what forced us to move from CMake — generation times were surpassing 5 minutes, and this is not counting configuring step. * Because in CMake there are several target properties that may be affected by the "parent" (another target which added…

I'm not trying to overlook cmake's issues... However my first thought on reading about your problem is "you shouldn't do that anyway - write simple code that someone else can understand". A little complexity is sometimes needed, but if your generations times are more than 30 seconds you probably should step back and do something different anyway.

Re: Faster Than Ninja

#39

Post by vient »

bluGill wrote: post #38
vient wrote: post #29

Earlier quoted context omitted.

Not only is CMake single threaded but the generation step time scales almost quadratically* with the number of targets in some cases. Probably not what happens here but this is what forced us to move from CMake — generation times were surpassing 5 minutes, and this is not counting configuring step. * Because in CMake there are several target properties that may be affected by the "parent" (another target which added…

I'm not trying to overlook cmake's issues... However my first thought on reading about your problem is "you shouldn't do that anyway - write simple code that someone else can understand". A little complexity is sometimes needed, but if your generations times are more than 30 seconds you probably should step back and do something different anyway.

Important detail here is that the problem disappers if you use PRIVATE dependencies — but you can't do that with a heavily templated C++ codebase. Templates (and complexity they bring) are there for good reasons (primarily performance at all costs) so we did not manage to find a path forward with CMake.

Re: Faster Than Ninja

#40

Post by oso2k »

Orphis wrote: post #31
evmar wrote: post #28

Earlier quoted context omitted.

I did some exploration of this idea in a followup build system! See https://neugierig.org/software/blog/2022/03/n2.html . (It's not really production-ready.)

Why not just embrace Content Based Addressing and the Bazel action cache? Not necessarily adopt the RBE protos, but you could have on-disk cache that are similar or even identical. Nowadays, I think a build tool that doesn't natively support distributed caching (and possibly remote execution) is a weird choice. And I don't think that spawning layers of processes allows for good parallelization as you don't know if an…

Sadly, I think things like distcc or ccache have mostly gone out of mode. Vertically scaling (multi core, multi thread, multi TB RAM build machines) and simple artifact caching have won for now. Artifact caching with dependency tracking in the midst of job scheduling was solved “good enough” by OS based mechanisms. Having used and managed the basics of a build farm, removing the need for networking, build farm management and build job coordination is a huge burden relief when your source tree doesn’t require it.

Part of the motivation to use processes is because their structure helps to keep the job generic, uni

Post reply on HN