Just use Bazel with remote execution. It can distribute your unit tests, too.
And cache build outputs: https://docs.bazel.build/versions/master/remote-caching.html
Distcc – distribute builds across multiple machines simultaneously (2006)
11–20 of 27 posts
Re: Distcc – distribute builds across multiple machines simultaneously (2006)
#12If you like distcc, perhaps you'd also like icecream https://github.com/icecc/icecream which I think is a bit easier to use.
I'm using it. It's reaction to network degradation, even one just moderately worse off than LAN with a single switch really hampers usage on cloud servers.
Re: Distcc – distribute builds across multiple machines simultaneously (2006)
#13Is distcc actually any faster than running the builds locally on a half decent machine? If I recall the latency associated with getting the file to the build machines often exceeded the local compilation duration. I remember being really excited about the idea, and really disappointed with the results.
One use that I can think of (never tried it myself, though in theory it should work) is if you are trying to cross compile for a processor architecture that doesn't have decent speed (many embedded chips), and the package you are compiling is difficult to cross compile (where the build script makes decisions based on the local environment or is otherwise to get to play nice with a cross compiler). In that case you ca…
Indeed, in my use (with the Parabola GNU/Linux-libre distro), this is the killer use-case of distcc. This is how most of Parabola's packages for MIPS were built, and how many for ARM are built.
Re: Distcc – distribute builds across multiple machines simultaneously (2006)
#14Earlier quoted context omitted.
Running it locally will always be faster as long as your machine is not a bottleneck (#cores, ram, ...). I think the use-case for distcc et al is to enable less-powerful machines to run builds faster by levering other machines. That’s exactly what we use it for at work. Our developers have not-so-powerful laptops and with distcc/icecc they can utilize the power of our build agents in the server room. Also interesting…
Or maybe if money is no object spin up a bunch of VMs in the cloud to compile.
At the moment, icecream is not suitable for this as it is super sensitive to network degradation, and I ran into few compilation stalls that way.
Re: Distcc – distribute builds across multiple machines simultaneously (2006)
#15Just use Bazel with remote execution. It can distribute your unit tests, too.
Not the same thing. distcc is a wrapper to a compiler that can work with any build system. Just use CC=distcc.
Re: Distcc – distribute builds across multiple machines simultaneously (2006)
#16Is distcc actually any faster than running the builds locally on a half decent machine? If I recall the latency associated with getting the file to the build machines often exceeded the local compilation duration. I remember being really excited about the idea, and really disappointed with the results.
We used it for several massive C++ apps ~2012 and yes it was much faster than our local machines, although our machines at the time weren't state of the art but also weren't bad either. We had a pretty amazing on-prem lab full of Blades that we used for distcc, so I'm sure that helped deliver the lightning.
One thing people always overlook though with distcc is that while you compile remote, you always link local. That was the bulk of our build time and wasn't the fault of distcc. Object files would come at lightning speed (compared to pure local build) but linking was still non-trivial.
These days I'd bet a modern AMD would beat distcc in our situation due to latency times.
Re: Distcc – distribute builds across multiple machines simultaneously (2006)
#17Just use Bazel with remote execution. It can distribute your unit tests, too.
Not the same thing. distcc is a wrapper to a compiler that can work with any build system. Just use CC=distcc.
I don’t think anyone is claiming that Bazel is “the same thing” as distcc. But they both solve the problem of distributed builds.
Re: Distcc – distribute builds across multiple machines simultaneously (2006)
#18If you like distcc, perhaps you'd also like icecream https://github.com/icecc/icecream which I think is a bit easier to use.
I worked on adding distributed compilation to sccache [0]. Docs at [1] and [2]. Compared to existing tools, sccache supports:
- local caching (like ccache)
- remote caching e.g. to S3, or a LAN redis instance (unique afaik)
- distributed compilation of C/C++ code (like distcc, icecream)
- distributed compilation of Rust (unique afaik)
- distributed compilation on Windows by cross compiling on Linux machines (unique afaik)
Note that I think bazel also does a bunch of these, but you need to use bazel as your build system.
[0] https://github.com/mozilla/sccache
[1] quickstart - https://github.com/mozilla/sccache/blob/master/docs/Distribu...
[2] reference docs - https://github.com/mozilla/sccache/blob/master/docs/Distribu...
Re: Distcc – distribute builds across multiple machines simultaneously (2006)
#19Is distcc actually any faster than running the builds locally on a half decent machine? If I recall the latency associated with getting the file to the build machines often exceeded the local compilation duration. I remember being really excited about the idea, and really disappointed with the results.
I was actually just playing around with distcc for the first time last night. Compiling ungoogled chromium normally takes my desktop about 12 hours, and using distcc to share the load with my laptop (with a gigabit connection to my desktop) took a little under 7 hours. It definitely improved the speed considerably for me.
Re: Distcc – distribute builds across multiple machines simultaneously (2006)
#20Is distcc actually any faster than running the builds locally on a half decent machine? If I recall the latency associated with getting the file to the build machines often exceeded the local compilation duration. I remember being really excited about the idea, and really disappointed with the results.
We used to use distcc (2007 time period) for the daily builds of one of our large, in-house C++ products - order of 10M LOC. In principle, it was a good use-case with a highly modular structure and a clearly defined but chunky build graph. In practice, it did work, but throwing more hardware at the problem on a single host turned out to be faster than the existing distcc setup and had much reduced operational complex…
This lines up with my experiences using it last time.