Parallel GCC: a research project aiming to parallelize a real-world compiler • Hacker News Board

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#71

Post by boring_twenties » Tue, Sep 17, 2019, 2:19 AM UTC

IcePic wrote: post #62

cat199 wrote: post #59

Earlier quoted context omitted.

> it sends preprocessed files just to not depend on remote env. to then be compiled by the host compiler ..

Sure, one compiler can't be egcs 2.95 and the other clang 8 (at least not in most cases) but the target compiler system doesn't even have to have include files, libs, libfoo-devel packages installed or they can be, but from old versions. As a proof of concept I once had a linux with gcc4.2.1 make c->.o files that later ran on OpenBSD. Would I trust that resulting binary? Sure not. But in cases of "lets spin up 10 com…

This doesn't work for C++, as there is no standard ABI and thus no safe way to link object files from different compiler versions into the same binary.

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#72

Post by Eug894 » Fri, Sep 20, 2019, 1:44 PM UTC

exikyut wrote: post #53

nwmcsween wrote: post #50

OT but why isn't there a compiler that simply does absolutely basic passes to get the IR into a 'normalized' form then apply optimizations based on a 'database' of super optimized 'chunks'? For unoptimized parts run the super optimizer.

So basically a code lowering engine that works a bit like https://en.wikipedia.org/wiki/Hashlife . Interesting! (NB. Googling "lowering" turned up the seemingly-random https://news.ycombinator.com/item?id=14422944 ; that page isn't such a bad set of starting links, so I figured I'd include it.)

Please, read this idea regarding use of superoptimizers: https://drive.google.com/file/d/1GSv89tiQmPDcnFEu4n4CqfaJcUJ...

Re: Parallel GCC: a research project aiming to parallelize a real-world compiler

#73

Post by Joky » Sun, Sep 22, 2019, 4:12 AM UTC

daddypro wrote: post #70

Joky wrote: post #44

Earlier quoted context omitted.

In LLVM, even with LLVM_ENABLE_THREADS, the compilation of a module is still sequential as far as I know. The LLVM Context is the unit of isolation between threads. For instance the multi-threaded ThinLTO optimizer/codegen will use one LLVM Context per-thread, and so each thread is processing a single Module / TU. The issue for concurrency is fairly deep in LLVM (use-lists, constant stored uniquely in context, etc.)…

Could you elaborate on what MLIR has done differently wrt making it multithread safe and what complications it's causing and what the trade offs are..

Sure: for instance in LLVM global values (variables, functions) have use-lists, so do constants. That means you can find easily every uses of a function: they are chained in a doubly linked list. In MLIR, it isn't the case. We use symbol name to refer to other global object, we lose the ability to easily find the uses of these global objects. On the other hand functions are well isolated and you can run Function passes in a multi-threaded fashion safely.