Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel GCC: a research project aiming to parallelize a real-world compiler

gcc.gnu.org

Post reply on HN
Print view View original

21–30 of 73 posts

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#21

Post by marmaduke »

saurik wrote: post #20
pjc50 wrote: post #3

The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".

That's great for super large builds, but the point of this is to deal with "I have one file I want to compile which is taking 30 seconds to compile, and I only have like three files to compile, so it would be awesome if I could use all of the cores on my laptop to do this compile and get it down to 10 seconds or something".

This, parti ularly for template heavy C++.

Older versions of Stan on GCC 4.8 would take 60+ seconds to compile a single C++ file.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#22

Post by modeless »

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over.

Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile your whole project, even if the output ends up being bit-identical.

Zapcc[1] is more advanced, but still far from ideal. What I really want is a compiler that can incrementally update the binary as I type code, and can start from a snapshot that I download from someone else so that nobody ever needs to do a full recompile. It would require a radically different compiler infrastructure, but I don't see any reason why it would be impossible.

[1] https://github.com/yrnkrn/zapcc

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#23

Post by barrkel »

ef004 wrote: post #14
blattimwind wrote: post #12

Earlier quoted context omitted.

From my observations multi-threading the linker would be more useful, since for small changes in bigger projects the linking step usually takes a lot longer than compiling a couple units.

Ive never found linking to take more than 5 seconds with the standard linker which, but lld exists and its super fast in my experience. But i guess its only needed whon working on huge c++ like firefox or sth. like that. https://lld.llvm.org/

Linking is isomorphic to tracing, moving GC. Effectively, the linker relocates objects in memory and writes a freeze-dried "heap" for the OS loader (another linker) to page in later.

Linker slowness ~= GC pause. The cost is approximately proportional to the size of the traced graph.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#25

Post by kachurovskiy »

modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

First time I spend a day on a partial-compile-only bug I switch if off and it's good old full recompilation from that point :)

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#26

Post by earenndil »

ndesaulniers wrote: post #6

I look forward to this. One that that will be important for reproducible builds is having tests for non determinism. Having nondeterministic code gen in a compiler is a source of frustration and dispair and sucks to debug.

My understanding from the article is that the code gen will still be deterministic--independent operations will be performed in parallel instead of in sequence, but data dependency will still be respected.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#27

Post by fanf2 »

modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

There is an interview with Anders Hejlsborg https://youtu.be/wSdV1M7n4gQ where he talks about how the Roslyn and TypeScript compilers are designed to support IDEs, with a structure that allows the compiler’s model of the program to be incrementally updated as the code is modified. So there has been some fairly prominent work along the lines you want...

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#28

Post by anaphor »

modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

You need something like http://adapton.org/ for C/C++ compilers really

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#29

Post by IcePic »

pstrateman wrote: post #5
pjc50 wrote: post #3

The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".

distcc can be useful, but requires that the local build environment matches the remote 100%.

No it doesn't, it sends preprocessed files just to not depend on remote env.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#30

Post by tempguy9999 »

kachurovskiy wrote: post #25
modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

First time I spend a day on a partial-compile-only bug I switch if off and it's good old full recompilation from that point :)

That happened to a colleague of mine with the Eiffel compiler (a long time ago). He was pissed off. Ditto full compilation thereafter.

Had something similar in our office with scala & SBT, but SBT is a POS anyway.

Post reply on HN