Live data from Hacker News

Working with jumbo/unity builds in C/C++

austinmorlan.com

31–40 of 45 posts

Re: Working with jumbo/unity builds in C/C++

#31
post #17
post #11

Earlier quoted context omitted.

Unless your amalgamated source file is getting into the 100s of megabytes to gigabytes range, I doubt that would be an issue.

If template instantiations use up AST node identifiers (I don't know whether whether they do), I can see scenarios where crazy amplified template instantiation chains can eat substantial chunks of that range.

They do.

Re: Working with jumbo/unity builds in C/C++

#32
post #27
post #23

> It's unfortunate that "Unity Build" is the prevailing term because it's impossible to do a search without getting a lot of results about the Unity game engine. I ignored this article on the front page because of that. Only because it stayed on the front page for several hours did (and because I care about C++ build issues) did I eventually click on it.

> and because I care about C++ build issues Look no further, these builds will give you more than enough issues on any sizable project.

Yes, glad I eventually clicked on it!

Re: Working with jumbo/unity builds in C/C++

#33

It's only faster if you are building from scratch, and even then it might not be faster. The resources required to build one huge compilation unit are also higher than compiling separately. If you support this type of build, it should not be the only option.

It is always faster in all cases from my experience. It should not, but it is. Try.

Re: Working with jumbo/unity builds in C/C++

#34
post #15

Unreal Engine does unity builds quite well - it will, as part of a prebuild step: 1. Merge related cpp and h files together in groups into monolith files, usually in the order of 10-20 source files merged based on my observations - often entire modules will be grouped together. 2. Exclude any individual files from the monolith that have edits since the last change. It's a nice middle ground, you don't substantially s…

Unreal's unity builds get very annoying as soon as your teams exceeds a handful of developers. You often have code that has builds fine on a machine but doesn't on another developer - it completely depends on what order the UBT picks. This is even more annoying when you have some people on Windows, some on Linux, some on Macs, and they consistently get different results. The fact that most tools (Intellisense, all cl…

I agree with your point on the C++ standards, but disagree about your complaints with Unreal's unity issues. There are issues with non-reproducibility in theory, but I've worked on massive projects and the impact is minimal.

> We worked around this by introducing a mandatory pre-merge CI stage that constantly does non-unity builds - but it's costly and not something a small company can often afford (a non-unity build of our UE project is ~20min on a Linux runner, and way more on a Windows one. That adds up fast).

Or you could only build the files that have changed. Even the largest of large files are one minute compiles.

> Unreal itself hasn't been non-unity buildable for a very long time.

Unreal builds in non-unity just fine. There was definitely a time period where it _didn't_, but for the last few years it's been much better than that.

Re: Working with jumbo/unity builds in C/C++

#35

It's only faster if you are building from scratch, and even then it might not be faster. The resources required to build one huge compilation unit are also higher than compiling separately. If you support this type of build, it should not be the only option.

It is always faster in all cases from my experience. It should not, but it is. Try.

This is silly. If I’m only rebuilding 1 file out of a 200 file source base, I guarantee the non-unity build will be faster. Just the number of characters I have to tokenize in this thought experiment should be enough to convince. If this isn’t the case then you’re doing some sort of n^2 c++ boost every header includes most other headers shenanigans and you need to stop that rather than doing a unity build.

Re: Working with jumbo/unity builds in C/C++

#36

It's only faster if you are building from scratch, and even then it might not be faster. The resources required to build one huge compilation unit are also higher than compiling separately. If you support this type of build, it should not be the only option.

It is always faster in all cases from my experience. It should not, but it is. Try.

I've worked with unity builds a lot in my career. Rebuilding a unity module can be 90s-2minutes (plus linking), whereas a single file might only be 3-5 seconds on its own.

The best possibility is if your build system can detect which files have changed and exclude them from unity builds, as this gives you the "slow path" the first time you change a file, but from then on you get the fast path behaviour.

Re: Working with jumbo/unity builds in C/C++

#37
post #3

Unfortunately this isn't consistent with how these terms are typically used. This article only explains the simple case where your program is small enough that you can compile all source files at once. That's not realistic or practical for much larger projects. Both WebKit and Chromium support unity/jumbo builds. They combine around 20 source files at a time into a single compilation unit, which provides a reasonable…

> Making a unity / jumbo build work for a project with 10,000+ files and millions of lines of code is not simple at all

For years we rolled our own unity build, but now CMake supports it directly through CMAKE_UNITY_BUILD and CMAKE_UNITY_BUILD_BATCH_SIZE, making it straightforward to enable.

When first enabling it on a large project, you'll run into clashes where different files contain identically-named file-scoped function or variables. Sometimes this reveals copy-pasted code, where the fix is to refactor the duplicate code anyway. Other times you just pick more specific names to avoid the clash.

We find unity build gives a solid 3X build speedup. We haven't eliminated header files, and in fact keep one slow CI job building the code without unity to ensure our code still builds either way.

Re: Working with jumbo/unity builds in C/C++

#38
post #15

Unreal Engine does unity builds quite well - it will, as part of a prebuild step: 1. Merge related cpp and h files together in groups into monolith files, usually in the order of 10-20 source files merged based on my observations - often entire modules will be grouped together. 2. Exclude any individual files from the monolith that have edits since the last change. It's a nice middle ground, you don't substantially s…

Unreal's unity builds get very annoying as soon as your teams exceeds a handful of developers. You often have code that has builds fine on a machine but doesn't on another developer - it completely depends on what order the UBT picks. This is even more annoying when you have some people on Windows, some on Linux, some on Macs, and they consistently get different results. The fact that most tools (Intellisense, all cl…

Have you considered adding something like ccache to your compiler to speed up your CI?

Re: Working with jumbo/unity builds in C/C++

#39
post #15

Unreal Engine does unity builds quite well - it will, as part of a prebuild step: 1. Merge related cpp and h files together in groups into monolith files, usually in the order of 10-20 source files merged based on my observations - often entire modules will be grouped together. 2. Exclude any individual files from the monolith that have edits since the last change. It's a nice middle ground, you don't substantially s…

You mean it does unreal builds.

Re: Working with jumbo/unity builds in C/C++

#40
post #7
post #3

Unfortunately this isn't consistent with how these terms are typically used. This article only explains the simple case where your program is small enough that you can compile all source files at once. That's not realistic or practical for much larger projects. Both WebKit and Chromium support unity/jumbo builds. They combine around 20 source files at a time into a single compilation unit, which provides a reasonable…

Firefox uses “unified” builds, concatenating .cpp files only within each directory. Unified Firefox builds are about 2-5x faster on my Mac. Non-unified builds are still built to make sure there are no unexpected side effects or accidental header file dependencies. https://firefox-source-docs.mozilla.org/build/buildsystem/un...

WebKit calls them that too.
Post reply on HN