Distcc: A fast, free distributed C/C++ compiler

We used to use this in a previous company. It reduced the build time from ~20 minutes to ~2 minutes; until a bug in the ld linker at the time (linking is not distributed - only compilation is) pushed it back up to 20 minutes. We had a hell of a time finding the issue, but luckily there was a newer versions of binutils available where it was fixed; and upgrading to it got the build time back under 2 minutes.

Fun times!

IIRC, when using distcc you must make sure that the Toolchain is ABI-perfect identified by its path.

So:

/opt/ourcompany/dev/bin/cc

absolutely *needs* to be the same on all machines involved or you risk very hard to spot issues.

For that reason, either use the absolute same distribution for everyone (and then run /use/bin/cc but watch out for alternatives!) or roll-out your own toolstack but make sure to put its version in the path.

Is bazel build distributed? distcc lets you use a lot of machines to speed up the wallclock-time of your compilation.

Back at Uni (two-and-a-half decades ago now), I setup a hacky version of this sort of thing to distribute compilations over several workstations:

* use make -j to run multiple task at once

* replace¹ gcc with a script that picked a host and ran the task on that²

This had many limitations that distcc lists it doesn't have (the machines had to have the same everything installed, the input and output locations for gcc had to be on a shared filesystem mounted in the same place on each host, it only parallelised the gcc part not linking or anything else, and it broke if any make steps had side-effects like making environment changes, etc.) and was a little fragile (it was just a quick hack…), but it worked surprisingly well overall. For small processes like our Uni work it didn't make much difference over make -j on its own³ but for building some larger projects like libraries we were calling that were not part of the department's standard Linux build, it could significantly reduce build times. On the machines we had back then a 25% improvement⁴ could mean quite a saving in wall-clock time.

One issue we ran into using it was that some makefiles were not well constructed in terms of dependency ordering, so would break⁵ with make -j even without my hack layered on top. They worked reliably for sequential processing which is presumably the only way the original author used them.

----

[1] well, not actually replace, but arrange for my script to be ahead of the real thing in the search path

[2] via rlogin IIRC, this predates SSH (or at least my knowledge of it)

[3] despite the single-core CPUs we had at the time, task concurrency still helped on a single host as IO bottlenecks meant that single core was less aggressively used without -j2 or -j3

[4] we sometimes measured noticeably more than that, but it depended on how patballisable the build actually was, and the shared IO resource was also shared by many other tasks over the network so that was a key bottleneck

[5] sometimes intermittently, so it could be difficult to diagnose

Assuming you don't use NFS to continuously read and write, I think it would be fine. Compiling is generally speaking CPU bound. Even the website mentions that it just copies all preprocessed code to the workers, so I expect this to be a one-time overhead per job.