Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Working with jumbo/unity builds in C/C++

austinmorlan.com

Post reply on HN
Print view View original

41–45 of 45 posts

Re: Working with jumbo/unity builds in C/C++

#41

Post by gpderetta »

npalli wrote: post #6
dmazzoni wrote: post #3

Unfortunately this isn't consistent with how these terms are typically used. This article only explains the simple case where your program is small enough that you can compile all source files at once. That's not realistic or practical for much larger projects. Both WebKit and Chromium support unity/jumbo builds. They combine around 20 source files at a time into a single compilation unit, which provides a reasonable…

Making a unity / jumbo build work for a project with 10,000+ files and millions of lines of code is not simple at all. True, but the number of such large projects is tiny (< 100?) compared to the tens of thousands of other smaller projects that might benefit. Good writeup.

I worked at companies that had more than 100s of code bases each over a milion lines of c++.

Re: Working with jumbo/unity builds in C/C++

#42

Post by qalmakka »

maccard wrote: post #34
qalmakka wrote: post #16

Earlier quoted context omitted.

Unreal's unity builds get very annoying as soon as your teams exceeds a handful of developers. You often have code that has builds fine on a machine but doesn't on another developer - it completely depends on what order the UBT picks. This is even more annoying when you have some people on Windows, some on Linux, some on Macs, and they consistently get different results. The fact that most tools (Intellisense, all cl…

I agree with your point on the C++ standards, but disagree about your complaints with Unreal's unity issues. There are issues with non-reproducibility in theory, but I've worked on massive projects and the impact is minimal. > We worked around this by introducing a mandatory pre-merge CI stage that constantly does non-unity builds - but it's costly and not something a small company can often afford (a non-unity build…

> the impact is minimal.

yes, I must admit it's not the end of the world, but it tends to add an extra layer of gotchas on top of our already crufty legacy codebase (millions of lines of code, >100 devs on 3 different platforms). Which is something I'd rather not have to deal with, honestly.

> Or you could only build the files that have changed

In my experience the UBT is somewhat inconsistent with that. I've seen multiple times for instance Mac builds being borked on our main branch because a Windows developer pushed code that built absolutely perfectly on their machine.

> Unreal builds in non-unity just fine. There was definitely a time period where it _didn't_, but for the last few years it's been much better than that.

last time I tried 5.1 refused to build non-unity on Linux. Did they fix it with the version after that?

Re: Working with jumbo/unity builds in C/C++

#43

Post by qalmakka »

intelVISA wrote: post #26
qalmakka wrote: post #16

Earlier quoted context omitted.

Unreal's unity builds get very annoying as soon as your teams exceeds a handful of developers. You often have code that has builds fine on a machine but doesn't on another developer - it completely depends on what order the UBT picks. This is even more annoying when you have some people on Windows, some on Linux, some on Macs, and they consistently get different results. The fact that most tools (Intellisense, all cl…

May be worth rethinking your module arrangements - unity builds have been stable for a while.

Sadly our codebase is full of legacy nonsense - which makes modularizing it a chore to say the least.

Re: Working with jumbo/unity builds in C/C++

#44

Post by qalmakka »

dafelst wrote: post #19
qalmakka wrote: post #16

Earlier quoted context omitted.

Unreal's unity builds get very annoying as soon as your teams exceeds a handful of developers. You often have code that has builds fine on a machine but doesn't on another developer - it completely depends on what order the UBT picks. This is even more annoying when you have some people on Windows, some on Linux, some on Macs, and they consistently get different results. The fact that most tools (Intellisense, all cl…

Really? That hasn't been my experience at all. What engine version are you on? We target windows, linux, xsx and ps5 and have probably 15-20 programmers making contributions daily and probably only hit maybe one or two of these issues per week, and they rarely get checked in as we get them during our mandatory preflight build during code review, similar to what you describe. We run all the preflight on on-prem machin…

> We did a lot of work to modularize our codebase so maybe that is helping?

Almost definitely the case. We have 100+ devs on a >1M legacy codebase with a lot of cross dependencies, and it's incredibly troublesome without a strict pre-build stage on CI (on premise of course, it's too expensive to run all of that on AWS)

Re: Working with jumbo/unity builds in C/C++

#45

Post by mathstuf »

forrestthewoods wrote: post #30
mattnewport wrote: post #29

Earlier quoted context omitted.

Modules, standardised in C++20, are the official solution both for the standard library and for other code but they are not universally well supported by all major compilers and build systems yet. Transitioning existing code to use modules is also not entirely straightforward, though probably no more problematic than introducing unity builds.

> though probably no more problematic than introducing unity builds. A "Unity build" really just means typing #include "foo.cpp" a few times. It's trivial. Meanwhile, neither Clang nor GCC support standard library modules. They have only partial support for modules themselves. C++ module support is non-existent in almost all build systems. https://en.cppreference.com/w/cpp/compiler_support The idea of C++ modules is…

It's been a lot of chicken-and-egg, but C++ modules finally have momentum. FWIW, Clang does support `import std` with `libc++` now. While I've implemented CMake support for modules, I've worked on laying the groundwork for compilers to be able to provide the required information (P1689) and I've been involved in advocating support from a number of build systems (e.g., Bazel, xmake, Meson, Tup). Some have been more receptive and made progress; others have had less.

Yes, it's very late, but progress is being made.

Post reply on HN