Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Unity builds lurked into the Firefox Build System

serge-sans-paille.github.io

Post reply on HN
Print view View original

41–50 of 64 posts

Re: Unity builds lurked into the Firefox Build System

#41

Post by StellarScience »

We leverage many third party C++ libraries with complex templates, concepts, and constexpr expressions that seem to require lots of CPU to compile. We've found unity builds to be almost 3X faster, so we make it the default for both developer and CI jobs.

But we keep a separate CI job that checks the non-unity build, so developers have to add the right #include statements and can't accidentally reference file-scoped functions from other files. While working on a given library or project, developers often disable the unity build for just that project to reduce incremental build times. It seems to offer the benefits of both approaches.

Precompiled headers don't give nearly the same speedup. We're excited for C++ modules of course, but we're trying to temper any expectations that modules will improve build speed.

Re: Unity builds lurked into the Firefox Build System

#42

Post by pphysch »

Scubabear68 wrote: post #40

Headers and C style macros are probably the most unfortunate aspects of C (and by extension, C++). So many hacks in compilers to try to work around this. A shame there is no language level fix for this nonsense. Really wish there could be a C++—- that would improve on C in areas like this, and avoid all the incredible nonsense of C++. And no, not Rust or Go.

Headers (in new code) will hopefully become optional due to modules. That would be such a big boost to the language.

C Macros are pretty much considered code smell in C++, right?

Re: Unity builds lurked into the Firefox Build System

#43

Post by zyedidia »

Scubabear68 wrote: post #40

Headers and C style macros are probably the most unfortunate aspects of C (and by extension, C++). So many hacks in compilers to try to work around this. A shame there is no language level fix for this nonsense. Really wish there could be a C++—- that would improve on C in areas like this, and avoid all the incredible nonsense of C++. And no, not Rust or Go.

I think Hare (https://harelang.org/) might fit the bill: it retains the minimalism and simplicity of C, but fixes issues like this (and others). Unfortunately I don't think it's ready for real use yet, but I am keeping an eye on it.

Re: Unity builds lurked into the Firefox Build System

#45

Post by tomjakubowski »

simplotek wrote: post #25
omoikane wrote: post #17

> This generally leads to faster compilation time in part because it aggregates the cost of parsing the same headers over and over. But this also reduces the opportunity to parallelize compilation across multiple files because they have been concatenated into fewer build units, and each unit now requires more memory to deal with the non-header parts. For some build systems and repositories, this actually increases bu…

> But this also reduces the opportunity to parallelize compilation across multiple files because they have been concatenated into fewer build units (...) Irrelevant. There is always significant overhead in handling multiple translation units, and unity builds simply eliminate that overhead. > and each unit now requires more memory to deal with the non-header parts. And that's perfectly ok. You can control how large u…

unity builds do often have worse performance than separate compilation for "incremental rebuilds" during development. that all depends on how the code is split up and how bad of a factor linking is.

as in the article, it's best to support both

Re: Unity builds lurked into the Firefox Build System

#46

Post by bgmeister »

thinkling wrote: post #19

I’ve been out of C/C++ development for a long time but seem to remember that precompiled headers were a thing back in the day. That approach didn’t have the name space issues pointed out here. Why are precompiled headers not used anymore?

They are still used in some places. But they have some downsides:

Precompiled headers don't play nicely with distributed compilation or shared build caches (which are perhaps the fastest way to build large C++ codebases). So while they can work well for local builds, they exclude the use of (IMO) better build-time optimisations.

They also require maintenance over time- if you precompile a bad set of headers it can make your compile times worse.

Re: Unity builds lurked into the Firefox Build System

#47

Post by pm215 »

10000truths wrote: post #39

With the advent of LTO, unity builds are mostly a band-aid for poor management of header files. The Linux kernel project was able to net a ~40% reduction in compilation CPU-time just by pruning the contents of some key header files [1]. It really boils down to two rules: 1. Don't declare anything in header files that is only used in one compilation unit. Internal structs and functions should be declared and defined i…

I think "just" is perhaps not the right word for something that took a senior dev over a year and more than 2000 commits just to get to an RFC patchset that doesn't compile for all architectures... Tremendous work, but it clearly wasn't easy or a matter of "follow these simple rules".

Re: Unity builds lurked into the Firefox Build System

#48

Post by pxeger1 »

Scubabear68 wrote: post #40

Headers and C style macros are probably the most unfortunate aspects of C (and by extension, C++). So many hacks in compilers to try to work around this. A shame there is no language level fix for this nonsense. Really wish there could be a C++—- that would improve on C in areas like this, and avoid all the incredible nonsense of C++. And no, not Rust or Go.

Have you tried Zig? I think it fits those criteria, and is known for its good build system, although AIUI it is quite a large language compared to C

Re: Unity builds lurked into the Firefox Build System

#49

Post by kccqzy »

To avoid some of these issues, it can be helpful in a project to require that all files including header files must be compilable on their own. Doesn't get rid of all the problems (you can still depend on transitive includes without explicitly including them) but enforces a minimal amount of code hygiene.

Re: Unity builds lurked into the Firefox Build System

#50

Post by maccard »

dagmx wrote: post #18
regnerba wrote: post #12

Earlier quoted context omitted.

Hahaha tell that to my Unreal Engine build times. A brand new AMD Epyc, 64 core machine, will take over an hour to compile. Good times.

I’d really like to see a comparison someday between Epics weird C# based build system and something like CMake+Ninja. I suspect there’s compilation optimizations to be made, but I don’t think it would save more than 30% here and there.

> I suspect there’s compilation optimizations to be made

There definitely are. I've spent a lot of time with UBT, and a "reasonable" amount of time with cmake and friends. UBT isn't quite the same as CMake + Ninja. UBT does "adaptive" unity builds, globbing, and a couple of other things.

> but I don’t think it would save more than 30% here and there.

Agreed. The clean build with UBT is painfully slow compared to Cmake + Ninja, but the full builds themselves are pretty good, and I'd bet that there's probably less low hanging fruit there.

I did a good chunk of work on improving compile times in Unreal, and there is definitely just low hanging fruit in the engine for improving compile times. Some changes to how UHT works around forward declares would also make a significant difference too.

Post reply on HN