Live data from Hacker News

Distcc: A fast, free distributed C/C++ compiler

distcc.org

81–90 of 103 posts

Re: Distcc: A fast, free distributed C/C++ compiler

#81
post #77

Nearly twenty years ago I had a little server farm of old PCs. Two or three Pentium-133s, one dual Pentium Pro 200 machine, and my pride and joy, a Pentium 3 running at 600 MHz. I was trying to get familiar with Gentoo and to make recompiling everything all the time more bearable I set up distcc so my P3 could do most of the work. It worked very well! But after a few weeks every Gentoo box in the house started crashi…

I still need to get around to setting up distcc; I only have two Gentoo servers, but one is so much more powerful than the other, and their CPUs are close enough I might be able to use ccache, too ...

Re: Distcc: A fast, free distributed C/C++ compiler

#82
post #47
post #45

Related: https://github.com/icecc/icecream - another option that does what distcc does, but aimed at a somewhat different use case. https://ccache.dev/ - a similar idea but provides caching of build outputs instead of distributing builds. You can use it together with distcc to achieve even better performance.

ccache is used together with distcc at the current place I am working at. Started digging at how these two work as I thought there is still room for improvement in our build times that can vary between 10 minutes to 1 hour. It is a huge code base, easily more than a million lines and around 18k files. But had to stop as there were way too many features to develop and bugs to fix. Also, management does not see that ki…

For builds that large, I (personally) start evaluating Bazel. Bazel has distributed build + shared cache features built-in. But I’ve always just dug into reducing build times in any large C or C++ code base I’ve worked on—damn what management says is important. And the switch to Bazel can be costly (effort) and it may be difficult to get team buy-in.

Re: Distcc: A fast, free distributed C/C++ compiler

#83
post #52
post #47

Earlier quoted context omitted.

ccache is used together with distcc at the current place I am working at. Started digging at how these two work as I thought there is still room for improvement in our build times that can vary between 10 minutes to 1 hour. It is a huge code base, easily more than a million lines and around 18k files. But had to stop as there were way too many features to develop and bugs to fix. Also, management does not see that ki…

You are probably aware, but for. others with ccache this is called "cache sloppiness", which is my favourite term. You can set this via config, as by default ccache is paranoid about being correct. But you can tweak it with things like setting a build directory home (this is great for me, as I'm the only user but compile things in say `/home/josh/dev/foo` and `/home/josh/dev/bar` and have my build directory as my dev…

The best use of distcc "at home" is when you have one or more "big iron" (desktop, server, whatever) and a few tiny machines that work just fine but don't have much processing power.

For example, with some work, you can setup distcc to cross-compile on your amd64 massive box for your raspberry pi.

Re: Distcc: A fast, free distributed C/C++ compiler

#84

Earlier quoted context omitted.

Clearcase was utter crap. 6 hour code checkouts and 2 weeks to setup a new developer is a freaking joke. I literally did a conversion from Clearcase to git and reduced the setup time to 15 minutes and this is for a code base older than Clearcase is. Not to mention the absolutely bad design for handling merge conflicts (punt to human if more than 1 person touched a file seriously???)

If you're talking about Clearcase snapshot views, I agree they were garbage. And IIRC merging in a Clearcase snapshot view was also a hot mess. Snapshot views was a bold-on that we were forced to use in later years. TBH the migration to other VCSs was already underway by then in our company but snapshot views was the last straw for us. On the other hand Clearcase dynamic views were pretty awesome. You just needed to…

we got a bit closer with gitfs but nobody has really merged all the parts into a "it just works" setup.

https://wiki.archlinux.org/title/Gitfs

Re: Distcc: A fast, free distributed C/C++ compiler

#85
post #77

Nearly twenty years ago I had a little server farm of old PCs. Two or three Pentium-133s, one dual Pentium Pro 200 machine, and my pride and joy, a Pentium 3 running at 600 MHz. I was trying to get familiar with Gentoo and to make recompiling everything all the time more bearable I set up distcc so my P3 could do most of the work. It worked very well! But after a few weeks every Gentoo box in the house started crashi…

Yeah. Everyone I’ve talked to who has run a build cluster has recommended ECC for the build cluster, even if they’ve decided not to use ECC for other systems. Some people would run Hackintosh-like setups for macOS build clusters, just for the ECC. Reproducible builds are also a big win here.

Absolutely! This was a very educational experience.

Re: Distcc: A fast, free distributed C/C++ compiler

#86

Hi, distcc's original author here. It's really nice that people are still enjoying and using it 20 years later. I have a new project that is in a somewhat similar space of wrapping compilers: https://github.com/sourcefrog/cargo-mutants , a mutation testing tool for Rust.

Pretty sure I used cargo-mutants on a lark during Advent of Code a couple years ago. Caught a couple bugs with it. Good stuff. Thanks! :D

Re: Distcc: A fast, free distributed C/C++ compiler

#88
post #47
post #45

Related: https://github.com/icecc/icecream - another option that does what distcc does, but aimed at a somewhat different use case. https://ccache.dev/ - a similar idea but provides caching of build outputs instead of distributing builds. You can use it together with distcc to achieve even better performance.

ccache is used together with distcc at the current place I am working at. Started digging at how these two work as I thought there is still room for improvement in our build times that can vary between 10 minutes to 1 hour. It is a huge code base, easily more than a million lines and around 18k files. But had to stop as there were way too many features to develop and bugs to fix. Also, management does not see that ki…

My codebase is significantly larger than yours (mine's a mix of mostly-C++ & some C) — perhaps 10–12 million lines. Clean builds are ~10m; clean-with-ccache are ~2m; incremental are millisecond.

I know this probably won't help with your current project, but you should think of your compiler as an exotic virtual machine: your code is the input program, and output executable is the output. Just like with a "real" CPU, there are ways to write a program that are fast, and ways to write a program that are slow.

To continue the analogy: if you have to sort a list, use `qsort()`, not `bubble sort()`.

So, for C/++ we can order the "cost" of various language features, from most-expensive-to-least-expensive:

    1. Deeply nested header-only (templated/inline) "libraries";
    2. Function overloading (especially with templates);
    3. Classes;
    4. Functions & type definitions; and,
    5. Macros & data.
That means, if you were to look at my code-base, you'll see lots-and-lots of "table driven" code, where I've encoded huge swathes of business logic as structured arrays of integers, and even more as macros-that-make-such-tables. This code compiles at ~100kloc/s.

We don't use function-overloading: one place we removed this reduced compile times from 70 hours to 20 seconds. Function-overloading requires the compiler to walk a list of functions, perform ADL, and then decide which is best. Functions that are "just C like" require a hash-lookup. The difference is about a factor of 10000 in speed. You can do "pretend" function-overloading by using a template + a switch statement, and letting template instantiation sort things out for you.

The last thing is we pretty much never allow "project" header files to include each other. More importantly, templated types must be instantiated once in one C++, and then `extern`ed. This is all the benefit of a template (write-once, reuse), with none of the holy-crap-we're-parsing-this-again issues.

Re: Distcc: A fast, free distributed C/C++ compiler

#89
post #81
post #77

Nearly twenty years ago I had a little server farm of old PCs. Two or three Pentium-133s, one dual Pentium Pro 200 machine, and my pride and joy, a Pentium 3 running at 600 MHz. I was trying to get familiar with Gentoo and to make recompiling everything all the time more bearable I set up distcc so my P3 could do most of the work. It worked very well! But after a few weeks every Gentoo box in the house started crashi…

I still need to get around to setting up distcc; I only have two Gentoo servers, but one is so much more powerful than the other, and their CPUs are close enough I might be able to use ccache, too ...

> their CPUs are close enough I might be able to use ccache, too

It’s the compiler that needs to line up for that. But my recommendation is to install sccache which will figure it out for you.

Re: Distcc: A fast, free distributed C/C++ compiler

#90

Hi, distcc's original author here. It's really nice that people are still enjoying and using it 20 years later. I have a new project that is in a somewhat similar space of wrapping compilers: https://github.com/sourcefrog/cargo-mutants , a mutation testing tool for Rust.

Wow, it's already 20 years. Was setting up distcc in my first job, with ccache for armcc. Good times, great experience. This was a life-safer in terms of compilation speed (just needed wait 20 mins for consolidation and like the same or more for loading symbols to the debugger, which was bearable comparing to spending another 2 or 3 hours taken by compilation). Having the opportunity, would like to thank you!
Post reply on HN