Parallel GCC: a research project aiming to parallelize a real-world compiler • Hacker News Board

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#31

Post by seamyb88 » Sat, Sep 14, 2019, 7:16 PM UTC

blattimwind wrote: post #12

ndesaulniers wrote: post #7

Earlier quoted context omitted.

Which is not parallelization. There's always the question of should I parallelize my compilation at the compiler level, or the build level (ie. Multiple translation units in flight). I think there's room for both, so you hopefully can get faster incremental compiles of a few number of TUs, but still have the old data level parallelization of multiple TUs.

From my observations multi-threading the linker would be more useful, since for small changes in bigger projects the linking step usually takes a lot longer than compiling a couple units.

I also find that the linker is the issue. Yes for performance, but also for testing (to the point of failure) linux's 'out of memory' scenario (try make -j8 on the llvm source, for example).

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#32

Post by the8472 » Sat, Sep 14, 2019, 7:23 PM UTC

modeless wrote: post #22

There are so many tools to parallelize compilation, but comparatively little effort has gone into reducing the vast amounts of redundant work that compilers do. A staggeringly large percentage of compilation time is simply wasted duplicating work that was already done hundreds of times over. Tools like ccache are really primitive compared to what is possible. Change one bit in a header file and you have to recompile…

Another waste of resources is that global dead code elimination happens really late (LTO). So functions are compiled and then discarded much later instead of not compiling them in the first place.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#33

Post by rumanator » Sat, Sep 14, 2019, 7:25 PM UTC

NilsIRL wrote: post #24

The use case is really niche. The only time it comes in handy is when: 1. You have enormous files AND 2. You have a little number of files.

Yes, I agree. In a world where all projects are comprised of a myriad source files where each one takes a fraction of a second to compile, this kind of approach sounds a tad pointless.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#34

Post by ris » Sat, Sep 14, 2019, 7:57 PM UTC

NilsIRL wrote: post #24

The use case is really niche. The only time it comes in handy is when: 1. You have enormous files AND 2. You have a little number of files.

I'm not sure about that. The traditional way of exploiting parallelism for large builds is to work on a per-file basis. Want more parallelism? Fork more jobs. But this can fall apart in a couple of ways:

1. It can get extremely expensive in memory use. Your parallelism limit can be dictated by the memory your machine has. Spending your available cores over fewer simultaneous compilation units could also have benefits arising from data locality, even in situations where a machine's memory headroom isn't a concern.

2. Many large build processes have serialized steps which are currently unable to benefit from any form of parallelism. This is becoming even more the case with the rise of IPO & LTO.

But as ever, we'll probably have to wait to find out how large the practical benefits end up being from this work.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#35

Post by jingwen » Sat, Sep 14, 2019, 8:07 PM UTC

cromka wrote: post #19

ndesaulniers wrote: post #8

Earlier quoted context omitted.

Google has objfs, a authenticed network mounted drive that is an object file cache. Usually most of the tree has already been compiles so just your change needs to get recompiled and relinked. Further, there's then GOMA which is the distcc equivalent. I frequently build Android with 500 cores in the cloud. For that reason, I don't think Google engineers will ever focus on the compiler speed of llvm. That said, LLD (l…

Interesting. I searched for some more information on that but don't see anything except the GitHub repostoryitself. I wonder how would one go about integrating that into .Net stack.

Check out Bazel's remote execution feature and Google RBE (remote build execution) [1]. Bazel itself is language agnostic (language support comes in the form of rules).

(disclaimer: I'm an engineer on the Bazel team)

[1] https://docs.bazel.build/versions/master/remote-execution.ht...

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#36

Post by dunkelheit » Sat, Sep 14, 2019, 8:19 PM UTC

The_rationalist wrote: post #4

Llvm already has one process per compile unit I believe.

So at a project that I was working on one of the compilation units got so unwieldy that the last part of the build process was basically waiting for it alone to compile (it contained a lot of template instantiations). The solution was of course to manually "shard" it into several files even if semantically it didn't make much sense. Now, you might argue that it was our shitty code that caused this but surely this is a fairly simple thing for a compiler to do automatically and not require the user do it.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#37

Post by loeg » Sat, Sep 14, 2019, 8:43 PM UTC

Huh, I thought compilation was already quite parallel even on many core machines. I guess that's only true for larger software projects (or ones with many small compilation units, at least). The step that could use some TLC and increased parallelism is linking.

On the other hand, maybe this project will help improve whole-program LTO link time. I see that is mentioned as future work for this effort:

> Parallelize IPA part. This can also improve the time during LTO compilations

Big kudos to Giuliano Belinassi, who seems to be the one driving this effort.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#38

Post by namibj » Sat, Sep 14, 2019, 9:24 PM UTC

dunkelheit wrote: post #36

The_rationalist wrote: post #4

Llvm already has one process per compile unit I believe.

So at a project that I was working on one of the compilation units got so unwieldy that the last part of the build process was basically waiting for it alone to compile (it contained a lot of template instantiations). The solution was of course to manually "shard" it into several files even if semantically it didn't make much sense. Now, you might argue that it was our shitty code that caused this but surely this is…

Even better: throw differential dataflow machinery at all the template handling of C++ code, interprocedural constant peopagation (and similar steps), and of course the linking itself.

It's sad these foundations only seem to exist in rust.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#39

Post by stochastic_monk » Sat, Sep 14, 2019, 9:29 PM UTC

marmaduke wrote: post #21

saurik wrote: post #20

Earlier quoted context omitted.

That's great for super large builds, but the point of this is to deal with "I have one file I want to compile which is taking 30 seconds to compile, and I only have like three files to compile, so it would be awesome if I could use all of the cores on my laptop to do this compile and get it down to 10 seconds or something".

This, parti ularly for template heavy C++. Older versions of Stan on GCC 4.8 would take 60+ seconds to compile a single C++ file.

I write template heavy code. I’ve recently refactored template instantiation across many object files. It’s faster, but a lot of work. It’d be great if I didn’t have to.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#40

Post by mrich » Sat, Sep 14, 2019, 9:45 PM UTC

NilsIRL wrote: post #24

The use case is really niche. The only time it comes in handy is when: 1. You have enormous files AND 2. You have a little number of files.

It is quite a common case. First you do your scratch make with 800 jobs which maxes out your machine just from preprocessing :) Thereafter you only edit a small portion of the code and recompile a small number of files, which only uses a few cores. It's especially problematic for C++ with all the template instantiations. Some files in our codebase take 40+ sec to compile.