The compilation-unit-per-file model (and in fact the whole concept of linking) are a legacy incremental build solution for C which somehow metastasized into fundamental requirements of building software on current OSes. It is an atrocity and should be disavowed by all developers.
Unity builds lurked into the Firefox Build System
51–60 of 64 posts
Re: Unity builds lurked into the Firefox Build System
#52Compilation units are a relic of a time where computers only had a few KB of memory. At this point computers are fast enough and have enough memory to compile the whole thing in one go faster than whatever gains doing change detection and linking will have.
Yeah, no. To this day Firefox developers building Gecko need a beefy desktop machine to be able to do it in a reasonable amount of time. I could do a clean build in 6 minutes with a ThreadRipper whose cores were all pegged, but forget doing the same in under an hour on a laptop. And that was with unified builds enabled.
And more importantly, on that same machine the build would take more than ten minutes without unified builds.
Re: Unity builds lurked into the Firefox Build System
#53Been using unity builds forever. The trick is to also have a standard build and try to compile it every week or so to catch anything that might have been missed, like a source file that is missing a header and wont compile alone because it got the header through the unity build ordering.
Re: Unity builds lurked into the Firefox Build System
#54With the advent of LTO, unity builds are mostly a band-aid for poor management of header files. The Linux kernel project was able to net a ~40% reduction in compilation CPU-time just by pruning the contents of some key header files [1]. It really boils down to two rules: 1. Don't declare anything in header files that is only used in one compilation unit. Internal structs and functions should be declared and defined i…
> unity builds are mostly a band-aid for poor management of header files
That's what its always was about (to improve build times), better optimization is just a welcome side effect. But header hygiene is hard because the problem will creep back into the code base over time.
> The Linux kernel project was able to net a ~40% reduction in compilation CPU-time
Linux is a C codebase. Header hygiene is much easier in C, because C headers usually only contain interface declarations (usually at most a few hundred lines of function prototypes and struct declarations), while C++ headers often need to include implementation code inside template functions, or are plastered with inline functions (which in turn means more dependencies to include in the header). And even if the user headers are reasonably 'clean', they still often need to include C++ stdlib headers which then indirectly introduce the same problem.
For instance your point (2) only makes sense if this header doesn't need to include any of the C++ stdlib headers, which will add tens of thousands of lines of code to each compilation unit. For such cases you might actually make the problem worse by splitting big headers into many smaller ones.
PS: the most effective, but also most radical and controversial solution is also a very simple one: don't include headers in headers.
Re: Unity builds lurked into the Firefox Build System
#55> This generally leads to faster compilation time in part because it aggregates the cost of parsing the same headers over and over. But this also reduces the opportunity to parallelize compilation across multiple files because they have been concatenated into fewer build units, and each unit now requires more memory to deal with the non-header parts. For some build systems and repositories, this actually increases bu…
You also need to consider that (at least in C++), your own code is just a very small snippet dangling off the end of a very large included stdlib code block, and that's for each source file which needs to include any C++ stdlib header.
For instance, just including in a C++ source file adds nearly 20kloc of code to the compilation unit:
https://www.godbolt.org/z/56ncqEqYs
If your project has 100 source files, each with 100 lines of code but each file includes the header (assuming this resolves to 20kloc), you will compile around 2mloc overall (100 * 20100 = 2010000).
If the same project code is in a single 10kloc source file which includes , you're only compiling 30kloc overall (100 * 100 + 20000 = 30000).
In such a case (which isn't even all that theoretical), you are just wasting a lot of energy keeping all your CPU cores busy compiling a hundred times over, versus compiling once on a single core ;)
Re: Unity builds lurked into the Firefox Build System
#56I’ve been out of C/C++ development for a long time but seem to remember that precompiled headers were a thing back in the day. That approach didn’t have the name space issues pointed out here. Why are precompiled headers not used anymore?
In a project that already has good header hygiene, precompiled headers don't help much to speed up builds. They're just a bandaid when the situation is already completely out of control.
Re: Unity builds lurked into the Firefox Build System
#57Headers and C style macros are probably the most unfortunate aspects of C (and by extension, C++). So many hacks in compilers to try to work around this. A shame there is no language level fix for this nonsense. Really wish there could be a C++—- that would improve on C in areas like this, and avoid all the incredible nonsense of C++. And no, not Rust or Go.
> Headers and C style macros are probably the most unfortunate aspects of C (and by extension, C++).
Headers only became a massive problem in C++ because of templates and the unfortunate introduction of the inline keyword (which then unfortunately also slipped into C99, truly the biggest blunder of the C committee next to VLAs).
Typical C headers (including the C stdlib headers) are at most a few hundred lines of function prototypes and struct declarations.
Typical C++ headers on the other hand (include the C++ stdlib headers) contain a mix of declarations and implementation code in template and inline functions and will pull in tens of thousands of lines of code into each compilation unit.
This is also the reason why typical C projects compile orders of magnitude faster than typical C++ projects with a comparable line count and number of source files.
Re: Unity builds lurked into the Firefox Build System
#58Re: Unity builds lurked into the Firefox Build System
#59> This generally leads to faster compilation time in part because it aggregates the cost of parsing the same headers over and over. But this also reduces the opportunity to parallelize compilation across multiple files because they have been concatenated into fewer build units, and each unit now requires more memory to deal with the non-header parts. For some build systems and repositories, this actually increases bu…
Not necessarily - I've been prototyping a fork of tcc that does both. It's multi-threaded rather than multiprocess.
Re: Unity builds lurked into the Firefox Build System
#60Earlier quoted context omitted.
I’d really like to see a comparison someday between Epics weird C# based build system and something like CMake+Ninja. I suspect there’s compilation optimizations to be made, but I don’t think it would save more than 30% here and there.
> I suspect there’s compilation optimizations to be made There definitely are. I've spent a lot of time with UBT, and a "reasonable" amount of time with cmake and friends. UBT isn't quite the same as CMake + Ninja. UBT does "adaptive" unity builds, globbing, and a couple of other things. > but I don’t think it would save more than 30% here and there. Agreed. The clean build with UBT is painfully slow compared to Cmak…
The big issue, in addition to speed, I had with UBT was how difficult it was to debug when it did the wrong thing. Often this was when having to adopt new Xcode versions, where CMake gave a lot of escape hatches to adapt it whereas UBT required spelunking.
At some points, there’s multiple layers of historic cruft that just seem arcane.
Last year, epic released a video where an engineer went through it and even they hit points where they said: “I have no idea what this area of code does”