Distcc: A fast, free distributed C/C++ compiler • Hacker News Board

Re: Distcc: A fast, free distributed C/C++ compiler

#71

Post by sourcefrog » Thu, Jun 01, 2023, 2:24 PM UTC

pjc50 wrote: post #33

anonymousDan wrote: post #29

Can anyone explain the architecture/how it works at a high level? I get that it is distributed. Does it basically copy the complete source tree to every worker and have them compile some independent subset of the object files? Does performance scale linearly with the number of worker nodes?

Last time I looked, it basically ran per-file "cc -E" on the source machine to get a compilation unit (optionally checking for a ccache cache hit at this point), then piped the result to "cc" running on the target machine, and copied the resulting object file back. > Does performance scale linearly with the number of worker nodes Yes, for small N. Overall scaling was limited by how much "make -j" the source machine c…

This was the original approach, although later work added an optional "pump mode" in which headers are distributed: https://manpages.ubuntu.com/manpages/bionic/man1/distcc-pump...

Re: Distcc: A fast, free distributed C/C++ compiler

#72

Post by sourcefrog » Thu, Jun 01, 2023, 2:25 PM UTC

RustyRussell wrote: post #25

Wow, it's still active! I remember when MBP wrote it while we were working at OzLabs together. One day I'll get back to that rewrite of ccontrol using modern distcc's features...

Hi Rusty!

For a later hack in the same vein, check out https://github.com/sourcefrog/cargo-mutants

Re: Distcc: A fast, free distributed C/C++ compiler

#73

Post by projektfu » Thu, Jun 01, 2023, 2:30 PM UTC

donaldihunter wrote: post #43

25+ years ago, our company used Clearcase for version control and it's clearmake had distributed build capability. Clearcase used a multi version file system (MVFS) and had build auditing so clearmake knew exactly what versions of source files were used in each build step. It could distribute build requests to any machine that could render the same "view" of the FS. Even without distributed builds, clearmake could re…

Around that time I went down the rabbit hole of "what if I wanted to write a filesystem for Windows NT?" It turns out that, aside from network redirectors, Microsoft didn't think this was something people ought to do very often. The people at Atria (which got bought by Rational) who were working on ClearCase were pioneers, and they bought a source license from Microsoft to be able to do it. The result was that there was a lot of information from them on a mailing list about how do implement a filesystem. However, it was mostly incomprehensible to me, because kernel programming on NT is quite different from what one would expect coming from other backgrounds, including Windows 95, OS/2, and Linux.

Re: Distcc: A fast, free distributed C/C++ compiler

#74

Post by ilyt » Thu, Jun 01, 2023, 2:32 PM UTC

pjc50 wrote: post #31

iveqy wrote: post #28

I'm curious about the security implications with using distcc. Doesn't this mean that if one computer gets compromised, the attacker can run code on all other computers using distcc, or secretly inject malicious code in the build result. So using distcc means that all computers using it must be trusted. And that means that using it on "all developers computers to share the load" is good for performance but bad for se…

Everything on the same LAN should generally be treated as "compromised/not compromised" together. There's rarely just a compromise of one machine in the same way there's never just one cockroach. I'm not sure whether distcc affects reproducible builds? You could, in any case, have tighter controls on the release builds, which would be done on a CI machine before signing. (Back when I used distcc we didn't distribute…

> Everything on the same LAN should generally be treated as "compromised/not compromised" together. There's rarely just a compromise of one machine in the same way there's never just one cockroach.

That might've been true 10 years ago and still is in some case but I wouldn't assume that now, far more things run over encryption for example, or have firewall

Re: Distcc: A fast, free distributed C/C++ compiler

#75

Post by donaldihunter » Thu, Jun 01, 2023, 2:32 PM UTC

z29LiTp5qUC30n wrote: post #57

donaldihunter wrote: post #43

25+ years ago, our company used Clearcase for version control and it's clearmake had distributed build capability. Clearcase used a multi version file system (MVFS) and had build auditing so clearmake knew exactly what versions of source files were used in each build step. It could distribute build requests to any machine that could render the same "view" of the FS. Even without distributed builds, clearmake could re…

Clearcase was utter crap. 6 hour code checkouts and 2 weeks to setup a new developer is a freaking joke. I literally did a conversion from Clearcase to git and reduced the setup time to 15 minutes and this is for a code base older than Clearcase is. Not to mention the absolutely bad design for handling merge conflicts (punt to human if more than 1 person touched a file seriously???)

If you're talking about Clearcase snapshot views, I agree they were garbage. And IIRC merging in a Clearcase snapshot view was also a hot mess. Snapshot views was a bold-on that we were forced to use in later years. TBH the migration to other VCSs was already underway by then in our company but snapshot views was the last straw for us.

On the other hand Clearcase dynamic views were pretty awesome. You just needed to edit your view config spec and the view FS would render the right file versions immediately. No checkout required. There was even view extended naming to let you open multiple versions of the same file directly from any editor.

As for merging Clearcase set the gold standard for three-way automatic merges and conflict resolution that wasn't matched until git came along. It's still superior in one important way - Clearcase had versioned directories as well as files so you could commit a file move and someone else's changes to the same "element" in the original file location would be merged correctly after the move. No heuristics, just versioned elements with versioned directory tree entries. Backporting fixes was a breeze, even with a significant refactor in between.

Git more or less wins hands down today because it is fast, distributed and works on change sets. But something with git's storage model, a multi version file system and Clearcase's directory versioning would be an awesome VCS.

Re: Distcc: A fast, free distributed C/C++ compiler

#76

Post by ilyt » Thu, Jun 01, 2023, 2:33 PM UTC

_joel wrote: post #21

Used to use this with Gentoo when doing emerges. Reusing old P3s and stuff, those were the day.. Insert XKCD "Compiling..." meme :)

Same, whole office stringed together so I could compile gentoo on my 700MHz laptop in reasonable time.

Re: Distcc: A fast, free distributed C/C++ compiler

#77

Post by omerhj » Thu, Jun 01, 2023, 2:41 PM UTC

Nearly twenty years ago I had a little server farm of old PCs. Two or three Pentium-133s, one dual Pentium Pro 200 machine, and my pride and joy, a Pentium 3 running at 600 MHz. I was trying to get familiar with Gentoo and to make recompiling everything all the time more bearable I set up distcc so my P3 could do most of the work. It worked very well!

But after a few weeks every Gentoo box in the house started crashing regularly. It took me a while to figure out what was going on: one of the slower machines had developed a single-bit memory error and was sharing corrupted .so files with all other machines.

Re: Distcc: A fast, free distributed C/C++ compiler

#78

Post by justinsaccount » Thu, Jun 01, 2023, 2:47 PM UTC

Erlangen wrote: post #14

Isn't it a misnomer to call it a "compiler"? Even it's github README says otherwise, > distcc is not itself a compiler, but rather a front-end to the GNU C/C++ compiler (gcc), or another compiler of your choice. All the regular gcc options and features work as normal.

20+ years ago I used distcc + jpegtran to rotate a few thousand jpegs. From what I remember I just had to write the Makefile as you would normally.

I think the end result was that the transfer time at 10mbps speeds made it take longer than it would have just transforming locally, but it was neat to see it work!

Re: Distcc: A fast, free distributed C/C++ compiler

#79

Post by klodolph » Thu, Jun 01, 2023, 2:53 PM UTC

omerhj wrote: post #77

Nearly twenty years ago I had a little server farm of old PCs. Two or three Pentium-133s, one dual Pentium Pro 200 machine, and my pride and joy, a Pentium 3 running at 600 MHz. I was trying to get familiar with Gentoo and to make recompiling everything all the time more bearable I set up distcc so my P3 could do most of the work. It worked very well! But after a few weeks every Gentoo box in the house started crashi…

Yeah. Everyone I’ve talked to who has run a build cluster has recommended ECC for the build cluster, even if they’ve decided not to use ECC for other systems. Some people would run Hackintosh-like setups for macOS build clusters, just for the ECC.

Reproducible builds are also a big win here.

Re: Distcc: A fast, free distributed C/C++ compiler

#80

Post by Cieric » Thu, Jun 01, 2023, 2:54 PM UTC

While it's purpose is different it can be used to do distributed compiling, so I'll leave it here.

https://github.com/Overv/outrun

Since I was just going down this rabbit hole recently, I kind of wonder if it's possible to set the filesystem on something more like the BitTorrent protocol so things like the libraries/compilers/headers that are used during compilation dont all need to come from the main pc. It probably wouldn't be useful until you reached a stupid number of computers and you started reaching the limits of the Ethernet wire, but for something stupid that can run on a pi cluster it would be a fun project.