Parallel GCC: a research project aiming to parallelize a real-world compiler
1–10 of 73 posts
Re: Parallel GCC: a research project aiming to parallelize a real-world compiler
#2A tricky job to say the least
Re: Parallel GCC: a research project aiming to parallelize a real-world compiler
#3The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".
Re: Parallel GCC: a research project aiming to parallelize a real-world compiler
#4Llvm already has one process per compile unit I believe.
Re: Parallel GCC: a research project aiming to parallelize a real-world compiler
#5The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".
distcc can be useful, but requires that the local build environment matches the remote 100%.
Re: Parallel GCC: a research project aiming to parallelize a real-world compiler
#6I look forward to this. One that that will be important for reproducible builds is having tests for non determinism. Having nondeterministic code gen in a compiler is a source of frustration and dispair and sucks to debug.
Re: Parallel GCC: a research project aiming to parallelize a real-world compiler
#7Llvm already has one process per compile unit I believe.
Which is not parallelization. There's always the question of should I parallelize my compilation at the compiler level, or the build level (ie. Multiple translation units in flight). I think there's room for both, so you hopefully can get faster incremental compiles of a few number of TUs, but still have the old data level parallelization of multiple TUs.
Re: Parallel GCC: a research project aiming to parallelize a real-world compiler
#8The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".
Google has objfs, a authenticed network mounted drive that is an object file cache. Usually most of the tree has already been compiles so just your change needs to get recompiled and relinked.
Further, there's then GOMA which is the distcc equivalent. I frequently build Android with 500 cores in the cloud. For that reason, I don't think Google engineers will ever focus on the compiler speed of llvm. That said, LLD (llvm's linker) is the fastest linker in town by far.
Re: Parallel GCC: a research project aiming to parallelize a real-world compiler
#9The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".
Distcc is neat, and invaluable, but it's parallelism on a different level to this, which is introducing parallelism at the single-file compilation level. It would be interesting to see the overall performance impact of this in combination with parallelism on a build level. It could even be harmful (but I doubt it) due to resource contention.
Re: Parallel GCC: a research project aiming to parallelize a real-world compiler
#10Llvm already has one process per compile unit I believe.
Which is not parallelization. There's always the question of should I parallelize my compilation at the compiler level, or the build level (ie. Multiple translation units in flight). I think there's room for both, so you hopefully can get faster incremental compiles of a few number of TUs, but still have the old data level parallelization of multiple TUs.
LLVM has actually been working through multithreaded compilation for years.
It actually works just fine (IIRC), but occasionally breaks when people do silly things :)
I believe you just need to enable LLVM_ENABLE_THREADS in the config.