Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel GCC: a research project aiming to parallelize a real-world compiler

gcc.gnu.org

Post reply on HN
Print view View original

11–20 of 73 posts

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#11

Post by phkahler »

ndesaulniers wrote: post #7
The_rationalist wrote: post #4

Llvm already has one process per compile unit I believe.

Which is not parallelization. There's always the question of should I parallelize my compilation at the compiler level, or the build level (ie. Multiple translation units in flight). I think there's room for both, so you hopefully can get faster incremental compiles of a few number of TUs, but still have the old data level parallelization of multiple TUs.

I only see parallelism at this level useful for JIT compilation. Interpreted languages, shaders, and similar. Those are good uses.

Not real valuable for building big projects that can build in parallel at the file level already.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#12

Post by blattimwind »

ndesaulniers wrote: post #7
The_rationalist wrote: post #4

Llvm already has one process per compile unit I believe.

Which is not parallelization. There's always the question of should I parallelize my compilation at the compiler level, or the build level (ie. Multiple translation units in flight). I think there's room for both, so you hopefully can get faster incremental compiles of a few number of TUs, but still have the old data level parallelization of multiple TUs.

From my observations multi-threading the linker would be more useful, since for small changes in bigger projects the linking step usually takes a lot longer than compiling a couple units.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#13

Post by tacostakohashi »

blattimwind wrote: post #12
ndesaulniers wrote: post #7

Earlier quoted context omitted.

Which is not parallelization. There's always the question of should I parallelize my compilation at the compiler level, or the build level (ie. Multiple translation units in flight). I think there's room for both, so you hopefully can get faster incremental compiles of a few number of TUs, but still have the old data level parallelization of multiple TUs.

From my observations multi-threading the linker would be more useful, since for small changes in bigger projects the linking step usually takes a lot longer than compiling a couple units.

There exists a multi-threaded linker called gold.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#14

Post by ef004 »

blattimwind wrote: post #12
ndesaulniers wrote: post #7

Earlier quoted context omitted.

Which is not parallelization. There's always the question of should I parallelize my compilation at the compiler level, or the build level (ie. Multiple translation units in flight). I think there's room for both, so you hopefully can get faster incremental compiles of a few number of TUs, but still have the old data level parallelization of multiple TUs.

From my observations multi-threading the linker would be more useful, since for small changes in bigger projects the linking step usually takes a lot longer than compiling a couple units.

Ive never found linking to take more than 5 seconds with the standard linker which, but lld exists and its super fast in my experience. But i guess its only needed whon working on huge c++ like firefox or sth. like that.

https://lld.llvm.org/

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#15

Post by kardos »

blattimwind wrote: post #12
ndesaulniers wrote: post #7

Earlier quoted context omitted.

Which is not parallelization. There's always the question of should I parallelize my compilation at the compiler level, or the build level (ie. Multiple translation units in flight). I think there's room for both, so you hopefully can get faster incremental compiles of a few number of TUs, but still have the old data level parallelization of multiple TUs.

From my observations multi-threading the linker would be more useful, since for small changes in bigger projects the linking step usually takes a lot longer than compiling a couple units.

In a similar vein, not all units cost the same to compile -- even with many jobs in a make -j situation, toward the end of the build there may be only a few units still in progress. These lengthy 'stragglers' might benefit the most from this parallelism and consequently reduce total compile time. Quite looking forward to this project landing in a stable release.

On the topic of linking-- can you sub in the gold linker for faster linking in your projects?

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#16

Post by spork12 »

pstrateman wrote: post #5
pjc50 wrote: post #3

The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".

distcc can be useful, but requires that the local build environment matches the remote 100%.

That is not true

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#18

Post by coldtea »

>This page introduces the Parallel GCC -- a research project aiming to parallelize a real-world compiler. This can be useful in many-core machines where GNU Make itself can not provide enough parallelism

Huh? One wants a fast compiler even for a single compilation, not just for multiple ones that parallel make can handle as implied here...

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#19

Post by cromka »

ndesaulniers wrote: post #8
pjc50 wrote: post #3

The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".

Google has objfs, a authenticed network mounted drive that is an object file cache. Usually most of the tree has already been compiles so just your change needs to get recompiled and relinked. Further, there's then GOMA which is the distcc equivalent. I frequently build Android with 500 cores in the cloud. For that reason, I don't think Google engineers will ever focus on the compiler speed of llvm. That said, LLD (l…

Interesting. I searched for some more information on that but don't see anything except the GitHub repostoryitself. I wonder how would one go about integrating that into .Net stack.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#20

Post by saurik »

pjc50 wrote: post #3

The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".

That's great for super large builds, but the point of this is to deal with "I have one file I want to compile which is taking 30 seconds to compile, and I only have like three files to compile, so it would be awesome if I could use all of the cores on my laptop to do this compile and get it down to 10 seconds or something".
Post reply on HN