Parallel GCC: a research project aiming to parallelize a real-world compiler • Hacker News Board

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#41

Post by Animats » Sat, Sep 14, 2019, 9:52 PM UTC

kachurovskiy wrote: post #25

modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

First time I spend a day on a partial-compile-only bug I switch if off and it's good old full recompilation from that point :)

That's a big problem with build systems based on timestamps, not checksums. Also, unless the build system can see all the inputs to the compilation, there's the potential for trouble.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#42

Post by andyayers » Sat, Sep 14, 2019, 10:20 PM UTC

FWIW, MSVC has had this ability for a few years now. It helps with both normal compilation unit compiles and with link-time code generation. See for instance Bruce Dawson's notes: https://randomascii.wordpress.com/2014/03/22/make-vc-compile... .

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#43

Post by slacka » Sat, Sep 14, 2019, 10:26 PM UTC

kachurovskiy wrote: post #25

modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

First time I spend a day on a partial-compile-only bug I switch if off and it's good old full recompilation from that point :)

Why not just do a full, clean build when you hit strange bug? If your workflow is anything like mine, partial-compile and tools like ccache have saved months of hours I would have wasted waiting for the compiler.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#44

Post by Joky » Sat, Sep 14, 2019, 10:28 PM UTC

DannyBee wrote: post #10

ndesaulniers wrote: post #7

Earlier quoted context omitted.

Which is not parallelization. There's always the question of should I parallelize my compilation at the compiler level, or the build level (ie. Multiple translation units in flight). I think there's room for both, so you hopefully can get faster incremental compiles of a few number of TUs, but still have the old data level parallelization of multiple TUs.

LLVM has actually been working through multithreaded compilation for years. It actually works just fine (IIRC), but occasionally breaks when people do silly things :) I believe you just need to enable LLVM_ENABLE_THREADS in the config.

In LLVM, even with LLVM_ENABLE_THREADS, the compilation of a module is still sequential as far as I know.

The LLVM Context is the unit of isolation between threads. For instance the multi-threaded ThinLTO optimizer/codegen will use one LLVM Context per-thread, and so each thread is processing a single Module / TU.

The issue for concurrency is fairly deep in LLVM (use-lists, constant stored uniquely in context, etc.) that would make it really difficult to address intra-module parallelism.

MLIR for example is designed with this in mind and the pass-manager is already multi-threaded at every level of nesting (function passes would runs on two functions of the same Module in parallel). This is causing other complication/inefficiency in the infrastructure, I'm still not sure how much it is a good tradeoff, we'll see...

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#45

Post by snazz » Sat, Sep 14, 2019, 10:42 PM UTC

modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

I have no idea how they do it, but Common Lisp compilers like SBCL and CCL compile each function incrementally, as you write them, ready to be played with in the REPL. This seems like a complex task in a language that doesn’t have the idea of an “image” that Smalltalk and Lisp do.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#46

Post by NullPrefix » Sat, Sep 14, 2019, 10:54 PM UTC

slacka wrote: post #43

kachurovskiy wrote: post #25

Earlier quoted context omitted.

First time I spend a day on a partial-compile-only bug I switch if off and it's good old full recompilation from that point :)

Why not just do a full, clean build when you hit strange bug? If your workflow is anything like mine, partial-compile and tools like ccache have saved months of hours I would have wasted waiting for the compiler.

>when you hit strange bug

Any bug is strange until you figure it out.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#47

Post by carlmr » Sat, Sep 14, 2019, 11:00 PM UTC

Animats wrote: post #41

kachurovskiy wrote: post #25

Earlier quoted context omitted.

First time I spend a day on a partial-compile-only bug I switch if off and it's good old full recompilation from that point :)

That's a big problem with build systems based on timestamps, not checksums. Also, unless the build system can see all the inputs to the compilation, there's the potential for trouble.

That's why Ninja build is so amazing for C/C++. It can use the whole dependencies output from the compiler and knows exactly what to recompile, even if only a header changed.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#48

Post by ori_b » Sat, Sep 14, 2019, 11:04 PM UTC

carlmr wrote: post #47

Animats wrote: post #41

Earlier quoted context omitted.

That's a big problem with build systems based on timestamps, not checksums. Also, unless the build system can see all the inputs to the compilation, there's the potential for trouble.

That's why Ninja build is so amazing for C/C++. It can use the whole dependencies output from the compiler and knows exactly what to recompile, even if only a header changed.

...You know, you can do that with plain old make, too?

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#49

Post by imtringued » Sat, Sep 14, 2019, 11:36 PM UTC

modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

One can also stop using boost. The last C++ project I worked on spent at least 30% of the compile time parsing boost/filesystem headers. I don't know what type of crackpot designed boost/filesystem but when you trace the includes (and the included includes) then you will see a reasonable amount for your own application. Maybe up to 50 headers that are part of the actual project and then comes a list of hundreds of boost header files.

I then proceeded to simply copy the includes of a random .cpp file and put it into a new .cpp file with nothing but includes. I removed all of them one by one except boost/filesystem. Compile times didn't change and if they did then only by a few milliseconds. A single include for boost/filesystem took 2 seconds to compile. I didn't even instantiate any templates in that simplified .cpp file so in practice the overhead is even higher. Isn't that absolutely insane?

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#50

Post by nwmcsween » Sat, Sep 14, 2019, 11:48 PM UTC

OT but why isn't there a compiler that simply does absolutely basic passes to get the IR into a 'normalized' form then apply optimizations based on a 'database' of super optimized 'chunks'? For unoptimized parts run the super optimizer.