Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Propeller: Profile-guided, optimizing, large-scale LLVM-based relinker

github.com

Post reply on HN
Print view View original

21–28 of 28 posts

Re: Propeller: Profile-guided, optimizing, large-scale LLVM-based relinker

#21

Post by jcelerier »

saagarjha wrote: post #16
jcelerier wrote: post #15

Earlier quoted context omitted.

> Template instantiations are duplicated even for types where the instantiations could have identical code. compilers and linkers had identical code folding optimizations for a looooong time (20+ years for msvc, a decade or so for GNU gold (--icf option) and GCC (-fipa-icf))

I was going to reply with a similar comment, but I'm unable to actually get this to happen when I tried it now: https://godbolt.org/z/4VkEQc . Perhaps the optimization needs more work?

No, it's just that the assembly output of godbolt happens before these optimization phases.

On my machine I get no unsigned instantiation, only the signed one :

    $ objdump -D a.out | c++filt | grep foo         
     563: e8 21 01 00 00        callq  689 (int)>
     56c: e8 18 01 00 00        callq  689 (int)>
    0000000000000689 (int)>:
and the main :

    0000000000000560 :
     560: 53                    push   %rbx
     561: 31 ff                 xor    %edi,%edi
     563: e8 21 01 00 00        callq  689 (int)>
     568: 31 ff                 xor    %edi,%edi
     56a: 89 c3                 mov    %eax,%ebx
     56c: e8 18 01 00 00        callq  689 (int)>
     571: 0f b6 d3              movzbl %bl,%edx
     574: 48 8d 3d b1 01 00 00  lea    0x1b1(%rip),%rdi        # 72c 
     57b: 0f b6 f0              movzbl %al,%esi
     57e: 31 c0                 xor    %eax,%eax
     580: e8 cb ff ff ff        callq  550 
     585: 31 c0                 xor    %eax,%eax
     587: 5b                    pop    %rbx
     588: c3                    retq   
     589: 00 00                 add    %al,(%rax)
     58b: 00 00                 add    %al,(%rax)
     58d: 00 00                 add    %al,(%rax)

Re: Propeller: Profile-guided, optimizing, large-scale LLVM-based relinker

#23

Post by ddcc7 »

chmaynard wrote: post #2

Propeller looks very interesting. Judging from the number of commits, the project has been around a long while. Does anyone know its history?

It's a fairly recent project, you can see the original announcement from the team on the LLVM mailing list in September 2019: https://lists.llvm.org/pipermail/llvm-dev/2019-September/135...

Re: Propeller: Profile-guided, optimizing, large-scale LLVM-based relinker

#24

Post by Rusky »

cwzwarich wrote: post #12
saagarjha wrote: post #3

> It has scalability issues and to rewrite a binary with a ~300M text segment size I'm curious what kinds of tasks require a binary with a 300 MB text size…

> I'm curious what kinds of tasks require a binary with a 300 MB text size… C++ lends itself to bloated binaries. Template instantiations are duplicated even for types where the instantiations could have identical code. Compilers have traditionally favored overly aggressive inlining over whole-program optimization. Rust didn't solve the problem; it just duplicated the C++ approach, and made it worse by effectively co…

Rust still has some room to improve over C++ here. Because it knows up-front which operations generic code can use on its type parameters, it can do things like "polymorphization": https://github.com/rust-lang/rust/pull/69749

This means the compiler will automatically detect how much duplication is actually needed ahead of time, eliminating the need for after-the-fact identical code folding. (This should also drastically reduce compile times for some programs!)

And it could go even further than that. For example, some code could be duplicated based only on the size+alignment of its parameters (C#-style generics without a JIT). The analysis could be finer-grained than per-function, sharing parts of a function across instantiations (kind of like outlining).

With the detection happening this early in the pipeline, the compiler could even tell the user about it and guide them toward techniques that reduce the amount of duplication. (A different language could go further, of course.)

Re: Propeller: Profile-guided, optimizing, large-scale LLVM-based relinker

#25

Post by mwcampbell »

Joky wrote: post #10
saagarjha wrote: post #7

Earlier quoted context omitted.

Yes, I get that; I am curious what goes into a binary with 300 MB of code in it.

Google has very large applications, but also they are using static linking: imagine that you statically link all the possible dependencies in a single binary for each application. 300MB is not considered a large binary at Google actually, I don't know what is public but you may look at Table 2 in this publication for instance: https://www.researchgate.net/publication/314105281_ThinLTO_S... I repro here some data, the…

Is Google's rationale for using static linking at this massive scale published anywhere?

Re: Propeller: Profile-guided, optimizing, large-scale LLVM-based relinker

#26

Post by jlokier »

mwcampbell wrote: post #25
Joky wrote: post #10

Earlier quoted context omitted.

Google has very large applications, but also they are using static linking: imagine that you statically link all the possible dependencies in a single binary for each application. 300MB is not considered a large binary at Google actually, I don't know what is public but you may look at Table 2 in this publication for instance: https://www.researchgate.net/publication/314105281_ThinLTO_S... I repro here some data, the…

Is Google's rationale for using static linking at this massive scale published anywhere?

I don't know Google's rationale, but static linked libraries usually run a little faster than dynamic linked ones.

Re: Propeller: Profile-guided, optimizing, large-scale LLVM-based relinker

#27

Post by fulafel »

Joky wrote: post #19
fulafel wrote: post #9

Can anyone summarize differences with previous work like https://cseweb.ucsd.edu/classes/sp00/cse231/dynamopldi.pdf ?

The closest previous work would rather be Facebook BOLT I believe: https://github.com/facebookincubator/BOLT

Right, I misremembered what Dynamo does - from the Bolt paper references I see that I was actually thinking of Spike (https://www.usenix.org/legacy/publications/library/proceedin...).

Interesting stuff!

Post reply on HN