Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Parallel GCC: a research project aiming to parallelize a real-world compiler

gcc.gnu.org

Post reply on HN
Print view View original

61–70 of 73 posts

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#61

Post by segfaultbuserr »

imtringued wrote: post #49
modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

One can also stop using boost. The last C++ project I worked on spent at least 30% of the compile time parsing boost/filesystem headers. I don't know what type of crackpot designed boost/filesystem but when you trace the includes (and the included includes) then you will see a reasonable amount for your own application. Maybe up to 50 headers that are part of the actual project and then comes a list of hundreds of bo…

> One can also stop using boost. The last C++ project I worked on spent at least 30% of the compile time parsing boost/filesystem headers.

I've seen one experimental IRC bot project that heavily uses template expansion, with the asynchronous model and other components in Boost, as a demonstration of advantages of modern C++. The whole compile requires 2 GiB+ memory, if multiprocess make is used, 4-8 GiB! The scale of the project is nowhere close to Chromium, building the Linux kernel doesn't need much RAM either, it's just a IRC bot! But the hardware requirements for using Boost and C++ template expansion is spectacular!

The actual binary runs well though, and does not need such hardware. And the developer simply saw it as a small price to pay for using these language features.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#62

Post by IcePic »

cat199 wrote: post #59
IcePic wrote: post #29

Earlier quoted context omitted.

No it doesn't, it sends preprocessed files just to not depend on remote env.

> it sends preprocessed files just to not depend on remote env. to then be compiled by the host compiler ..

Sure, one compiler can't be egcs 2.95 and the other clang 8 (at least not in most cases) but the target compiler system doesn't even have to have include files, libs, libfoo-devel packages installed or they can be, but from old versions. As a proof of concept I once had a linux with gcc4.2.1 make c->.o files that later ran on OpenBSD. Would I trust that resulting binary? Sure not.

But in cases of "lets spin up 10 compile boxes with " or have something along the lines of Xcode which allowed you and your colleagues to help out each other with small work units in order to make a single compile run quite much faster, that is a definite possibility.

For that final release build, you might consider running it on one of those 10 VMs in the example above and have 'only' 9 others help out with the sub-parts in order to get some kind of .. guarantee.

If it takes a minute for a full build on one box, getting a hint after 10 seconds that you misspelled something and that it won't ever link decently is worth something too if you value your time as a developer.

Not saying it's perfect, only chipped in on the "must be 100% or it can't ever work", hopefully without needing to go into the "coach says we need to give 110% this game, and 120% if it's the finals" nitpicking.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#63

Post by ndesaulniers »

tacostakohashi wrote: post #13
blattimwind wrote: post #12

Earlier quoted context omitted.

From my observations multi-threading the linker would be more useful, since for small changes in bigger projects the linking step usually takes a lot longer than compiling a couple units.

There exists a multi-threaded linker called gold.

LLD (llvm's linker) is also threaded but much faster than gold.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#64

Post by Const-me »

big_chungus wrote: post #51
imtringued wrote: post #49

Earlier quoted context omitted.

One can also stop using boost. The last C++ project I worked on spent at least 30% of the compile time parsing boost/filesystem headers. I don't know what type of crackpot designed boost/filesystem but when you trace the includes (and the included includes) then you will see a reasonable amount for your own application. Maybe up to 50 headers that are part of the actual project and then comes a list of hundreds of bo…

Much of boost's compilation time is a result of being so template-heavy. It's still got to parse and do much of the generic processing, even if you don't actually instantiate a template. Whether boost's style is the pinnacle of elegance or ugly as sin is still somewhat in contention, but no one wants to re-write it. I am some what put off as some one who started with C, but most of the C++ people reference it regular…

Microsoft's ATL and WTL are comparable to parts of standard library and boost. They build very fast, despite also based on templates. Often run much faster too, e.g. CAtlMap is typically faster than std::unordered_map by an order of magnitude, due to cache friendliness. The only downside, it's Windows only.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#65

Post by bgmeister »

pstrateman wrote: post #5
pjc50 wrote: post #3

The old-school approach to this was "distcc". At the company where we used it for C++ we had a small compile farm and usually did "make -j 50".

distcc can be useful, but requires that the local build environment matches the remote 100%.

Icecc (an old distcc fork) also distributes the toolchain, to avoid that particular problem.

https://github.com/icecc/icecream

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#66

Post by bgmeister »

jingwen wrote: post #35
cromka wrote: post #19

Earlier quoted context omitted.

Interesting. I searched for some more information on that but don't see anything except the GitHub repostoryitself. I wonder how would one go about integrating that into .Net stack.

Check out Bazel's remote execution feature and Google RBE (remote build execution) [1]. Bazel itself is language agnostic (language support comes in the form of rules). (disclaimer: I'm an engineer on the Bazel team) [1] https://docs.bazel.build/versions/master/remote-execution.ht...

There's also an opensource goma client + server which proxies to RBE (or potentially other bazel remote execution API implementations). Which is great if you can't or don't want to use the bazel client, but can use a compiler wrapper.

https://chromium.googlesource.com/infra/goma/client/ https://chromium.googlesource.com/infra/goma/server/

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#67

Post by joshuamorton »

jart wrote: post #56
exikyut wrote: post #54

Earlier quoted context omitted.

Nice info. I've wondered for quite a while about the impact of the G level of compile parallelization :). A couple of followup questions, if I may: - How long do incremental and non-incremental builds take with the build configuration you describe? - How does core engineer allocation work? Is 500 cores an average number? High-priority work presumably commands more access, but I'm curious if allocation is generally te…

Google has unlimited resources. Cores and codebase size don't matter to them. Only concern is what's on the critical path, e.g. final link step. It's very rare for builds to not be incremental. AFAIK JavaScript is the only language that can't be compiled incrementally. Probably the only engineers who would need to run a build solely on local hardware are ones that help support open source tools.

Js can and does get compiled incrementally if people do it correctly. It's fully supported by blaze. It's just that a bunch of old codebases globbed everything, so it appeared no incremental.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#68

Post by jart »

joshuamorton wrote: post #67
jart wrote: post #56

Earlier quoted context omitted.

Google has unlimited resources. Cores and codebase size don't matter to them. Only concern is what's on the critical path, e.g. final link step. It's very rare for builds to not be incremental. AFAIK JavaScript is the only language that can't be compiled incrementally. Probably the only engineers who would need to run a build solely on local hardware are ones that help support open source tools.

Js can and does get compiled incrementally if people do it correctly. It's fully supported by blaze. It's just that a bunch of old codebases globbed everything, so it appeared no incremental.

Is that with the js_binary/library rules or something new? Last time I checked, all the work that goes into compiling the JS sources and minifying them into a single "binary" blob happens in the js_binary definition, and the js_library definitions traditionally served only to construct a total ordering of raw source source file dependencies.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#69

Post by joshuamorton »

jart wrote: post #68
joshuamorton wrote: post #67

Earlier quoted context omitted.

Js can and does get compiled incrementally if people do it correctly. It's fully supported by blaze. It's just that a bunch of old codebases globbed everything, so it appeared no incremental.

Is that with the js_binary/library rules or something new? Last time I checked, all the work that goes into compiling the JS sources and minifying them into a single "binary" blob happens in the js_binary definition, and the js_library definitions traditionally served only to construct a total ordering of raw source source file dependencies.

Two partial answers: there's a newer better thing thats now the default and splits the building, but even under old js_library, there were some gains from not globbing everything together...

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#70

Post by daddypro »

Joky wrote: post #44
DannyBee wrote: post #10

Earlier quoted context omitted.

LLVM has actually been working through multithreaded compilation for years. It actually works just fine (IIRC), but occasionally breaks when people do silly things :) I believe you just need to enable LLVM_ENABLE_THREADS in the config.

In LLVM, even with LLVM_ENABLE_THREADS, the compilation of a module is still sequential as far as I know. The LLVM Context is the unit of isolation between threads. For instance the multi-threaded ThinLTO optimizer/codegen will use one LLVM Context per-thread, and so each thread is processing a single Module / TU. The issue for concurrency is fairly deep in LLVM (use-lists, constant stored uniquely in context, etc.)…

Could you elaborate on what MLIR has done differently wrt making it multithread safe and what complications it's causing and what the trade offs are..
Post reply on HN