91–100 of 103 posts
1) set up passwordless, ssh.
and
2) use the gnu parallel. https://www.gnu.org/software/parallel/
gnu parallel is super flexible, very useful.
Hi, distcc's original author here. It's really nice that people are still enjoying and using it 20 years later. I have a new project that is in a somewhat similar space of wrapping compilers: https://github.com/sourcefrog/cargo-mutants , a mutation testing tool for Rust.
So many memories. Back in grad school I would run this on a handful of workstations to speed up compilations. Really neat tool, and a clever setup.
Back when I ran Gentoo, distcc and ccache were pretty a necessity.
25+ years ago, our company used Clearcase for version control and it's clearmake had distributed build capability. Clearcase used a multi version file system (MVFS) and had build auditing so clearmake knew exactly what versions of source files were used in each build step. It could distribute build requests to any machine that could render the same "view" of the FS. Even without distributed builds, clearmake could re…
> I'd say set the gold standard for version control and dependency tracking and nothing today comes close to it.
In my 2005-2011 experience with Clearcase, it was slow and required dedicated developers just to manage the versions, and I'm so happy its version control model has been an evolutionary dead-end in the greater developer community. The MVFS is an attractive trap. Giving people the ability to trivially access older versions means you've just outsourced the job of keeping everything working together to some poor SOB. It was very much a "enough rope to hang yourself" kind of design.
As I said, it was slow, because MVFS. The recommended solution from Clearcase/IBM was to break up the source tree into different volumes (or whatever Clearcase's "repo"-analogue was named), which just increased the pain of keeping things in sync.
Additionally, it was an "ask-permission" design for modifying files, where you had to checkout a file before being able to modify it, and you couldn't checkout if someone else had, which added a ton of painful overhead.
I'll grant that my company/group didn't know what they were doing, but following IBM/Clearcase's guidance was not a good idea.
These days, I use Clearcase as a warning to the younger generation.
Earlier quoted context omitted.
ccache is used together with distcc at the current place I am working at. Started digging at how these two work as I thought there is still room for improvement in our build times that can vary between 10 minutes to 1 hour. It is a huge code base, easily more than a million lines and around 18k files. But had to stop as there were way too many features to develop and bugs to fix. Also, management does not see that ki…
My codebase is significantly larger than yours (mine's a mix of mostly-C++ & some C) — perhaps 10–12 million lines. Clean builds are ~10m; clean-with-ccache are ~2m; incremental are millisecond. I know this probably won't help with your current project, but you should think of your compiler as an exotic virtual machine: your code is the input program, and output executable is the output. Just like with a "real" CPU,…
The only downside is that it adds a ton of boilerplate and a lot of maintenance overhead. You need separate compilation units for everything and then you need a sub-struct to use the pimpl approach. Fast pimpl (in-place new in reserved space in the parent struct itself) gets rid of the heap allocations but you still have a pointer indirection and prevent the compiler from properly stripping out unused code across translation units normally (that’s where LTO comes in these days).
Really, the problem is just that it’s a PITA to write compared to sticking everything in the header file.
(It’s ironic that rust meets the first two rules by design but is still much slower than C++ to compile, though it does imply what’s already known, specifically that there’s a lot of room for improvement.)