So many memories. Back in grad school I would run this on a handful of workstations to speed up compilations. Really neat tool, and a clever setup.
Back when I ran Gentoo, distcc and ccache were pretty a necessity.
Distcc: A fast, free distributed C/C++ compiler
61–70 of 103 posts
Re: Distcc: A fast, free distributed C/C++ compiler
#62> distcc sends the complete preprocessed source code across the network for each job, s
I assume that for template-heavy c++, that could easily be hundreds of gigabytes for 1 gigabyte of c++ code to compile...?
If you're working from a laptop, surely the wifi connection will by far be the bottleneck?
Re: Distcc: A fast, free distributed C/C++ compiler
#63IIRC, when using distcc you must make sure that the Toolchain is ABI-perfect identified by its path. So: /opt/ourcompany/dev/bin/cc absolutely *needs* to be the same on all machines involved or you risk very hard to spot issues. For that reason, either use the absolute same distribution for everyone (and then run /use/bin/cc but watch out for alternatives!) or roll-out your own toolstack but make sure to put its vers…
Sounds like a great use case for containers, no?
Maybe, but you need to run the exact came container on everyone's machines.
Re: Distcc: A fast, free distributed C/C++ compiler
#64> distcc sends the complete preprocessed source code across the network for each job, s I assume that for template-heavy c++, that could easily be hundreds of gigabytes for 1 gigabyte of c++ code to compile...? If you're working from a laptop, surely the wifi connection will by far be the bottleneck?
It is. These systems (inctedibuild, fastbuild) tend to work extremely well but are network bottlenecked. You need a fast, low latency connection.
Re: Distcc: A fast, free distributed C/C++ compiler
#65I'm curious about the security implications with using distcc. Doesn't this mean that if one computer gets compromised, the attacker can run code on all other computers using distcc, or secretly inject malicious code in the build result. So using distcc means that all computers using it must be trusted. And that means that using it on "all developers computers to share the load" is good for performance but bad for se…
I don't know about other platforms... but on Gentoo, distcc is restricted to only running a small list of programs (ie gcc). That means you can't just send random commands to the distcc host from your "slow machines".
Granted, if your distcc host gets compromised, the compiled output shouldn't be trusted until the server can be reprovisioned.
Re: Distcc: A fast, free distributed C/C++ compiler
#66Hi, distcc's original author here. It's really nice that people are still enjoying and using it 20 years later.
I have a new project that is in a somewhat similar space of wrapping compilers: https://github.com/sourcefrog/cargo-mutants, a mutation testing tool for Rust.
Re: Distcc: A fast, free distributed C/C++ compiler
#67I'm curious about the security implications with using distcc. Doesn't this mean that if one computer gets compromised, the attacker can run code on all other computers using distcc, or secretly inject malicious code in the build result. So using distcc means that all computers using it must be trusted. And that means that using it on "all developers computers to share the load" is good for performance but bad for se…
Yes, that's correct: reusing developer's computers means you're trusting all of them. At small scale, that may be acceptable because you probably trust them all a lot anyhow, and perhaps are unlikely to have a strictly hermetic and isolated build system.
When the team is bigger or when security is more important, it's important to have a build system where you're confident that no one can subvert the output, and that includes ensuring that very few people can control machines running distccd.
Another way in which it shows its age is that, by default, there are only netblock based restrictions on clients, and connections are over unencrypted TCP by default (last time I looked), although there is an option to use SSH (or I guess Tailscale or similar.)
Re: Distcc: A fast, free distributed C/C++ compiler
#68Earlier quoted context omitted.
Is bazel build distributed? distcc lets you use a lot of machines to speed up the wallclock-time of your compilation.
> Is bazel build distributed? Yes, that's the main feature of Bazel. And it caches the generated files. Although you could do theoretically cache using ccache with distcc.
I doubt that it is the main feature. RBE is difficult to deploy and consequently rarely used. On the other hand of you assume that 99% of bazel invocations are done by google, then weighted that way RBE is nearly universal.
Re: Distcc: A fast, free distributed C/C++ compiler
#69The biggest problem with distcc is that it fails with architecture-specific instruction sets like avx. A problem not shared by icecc.
Isn’t that just a matter of setting appropriate target flags, and not letting them by implied? That’s basic build hygiene anyway.
Re: Distcc: A fast, free distributed C/C++ compiler
#70How do these distributed build tools work across the Internet, i.e. outside of a local LAN office setting? Does the increased latency and slower bandwidth become a bottleneck?
They will work over the internet. You should use SSH or a VPN like Tailscale because the native protocol is not encrypted.
Bandwidth and latency effects will have some effect on how well it will scale, but on a reasonably high latency and low RTT connection it's probably still faster than purely local compilation. Sending work over the network takes time but not much CPU, so the local CPU is freed up to do other work.