Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wild – A fast linker for Linux

github.com

Post reply on HN
Print view View original

31–40 of 222 posts

Re: Wild – A fast linker for Linux

#31

Post by 0x457 »

secondcoming wrote: post #27
pzmarzly wrote: post #19

Ever since mold relicensed from AGPL to MIT (as part of mold 2.0 release), the worldwide need for making another fast linker has been greatly reduced, so I wasn't expecting a project like this to appear. And definitely wasn't expecting it to already be 2x faster than mold in some cases. Will keep an eye on this project to see how it evolves, best of luck to the author.

Maybe I'm holding it wrong, but mold isn't faster at all if you're using LTO, which you probably should be.

I think we're talking about non-release builds here. In those, you don't want to use LTO, you just want to get that binary as fast as possible.

Re: Wild – A fast linker for Linux

#32

Post by [deleted] »

secondcoming wrote: post #27
pzmarzly wrote: post #19

Ever since mold relicensed from AGPL to MIT (as part of mold 2.0 release), the worldwide need for making another fast linker has been greatly reduced, so I wasn't expecting a project like this to appear. And definitely wasn't expecting it to already be 2x faster than mold in some cases. Will keep an eye on this project to see how it evolves, best of luck to the author.

Maybe I'm holding it wrong, but mold isn't faster at all if you're using LTO, which you probably should be.

[deleted]

Re: Wild – A fast linker for Linux

#33

Post by eseidel »

devit wrote: post #7

I think the optimal approach for development would be to not produce a traditional linked executable at all, but instead just place the object files in memory, and then produce a loader executable that hooks page faults in those memory areas and on-demand mmaps the relevant object elsewhere, applies relocations to it, and then moves it in place with mremap. Symbols would be resolved based on an index where only updat…

Sounds like Apple's old ZeroLink from the aughts?

Re: Wild – A fast linker for Linux

#34

Post by all2 »

fsfod wrote: post #30
devit wrote: post #7

I think the optimal approach for development would be to not produce a traditional linked executable at all, but instead just place the object files in memory, and then produce a loader executable that hooks page faults in those memory areas and on-demand mmaps the relevant object elsewhere, applies relocations to it, and then moves it in place with mremap. Symbols would be resolved based on an index where only updat…

You can sort of do that with some of LLVM's JIT systems https://llvm.org/docs/JITLink.html , I'm surprised that no one has yet made a edit and continue system using it.

My parens sense is tingling. This sounds like a lisp-machine, or just standard lisp development environment.

Re: Wild – A fast linker for Linux

#35

Post by cma »

cogman10 wrote: post #25
ComputerGuru wrote: post #16

Earlier quoted context omitted.

Actually a lot of the hacks that mold uses to be the fastest linker would be, ironically, harder to reproduce with rust because they’re antithetical to its approach. Eg Mold intentionally eschews used resource collection to speed up execution (it’ll be cleaned up by the os when the process exits) while rust has a strong RAII approach here that would introduce slowdowns.

Depends on how things are approached. You could, for example, take advantage of bump arena allocator in rust which would allow the linker to have just 1 alloc/dealloc. Mold is still using more traditional allocators under the covers which won't be as fast as a bump allocator. (Nothing would stop mold from doing the same).

Traditional allocators are fast if you never introduce much fragmentation with free, though you may still get some gaps and have some other overhead and not be quite as fast. But why couldn't you just LD_PRELOAD a malloc for mold that worked as a bump/stack/arena allocator and just ignored free if anything party stuff isn't making that many allocations?

Re: Wild – A fast linker for Linux

#36

Post by checker659 »

devit wrote: post #7

I think the optimal approach for development would be to not produce a traditional linked executable at all, but instead just place the object files in memory, and then produce a loader executable that hooks page faults in those memory areas and on-demand mmaps the relevant object elsewhere, applies relocations to it, and then moves it in place with mremap. Symbols would be resolved based on an index where only updat…

Linker overlays?

Re: Wild – A fast linker for Linux

#37

Post by Aurornis »

ComputerGuru wrote: post #26
Philpax wrote: post #21

Earlier quoted context omitted.

I mean, that's pretty easy to do in Rust: https://doc.rust-lang.org/std/mem/struct.ManuallyDrop.html Also see various arena allocator crates, etc.

Not really. You would have to either wrap any standard library types in newtypes with ManuallyDrop implemented or (for some) use a custom allocator. And if you want to free some things in one go but not others that gets much harder, especially when you look at how easy a language like zig makes it. And if you intentionally leak everything it is onerous to get the borrow checker to realize that unless you use a leaked…

> You would have to either wrap any standard library types in newtypes with ManuallyDrop implemented

ManuallyDrop would presumably be implemented on large data structures where it matters, not on every single type involved in the program.

Re: Wild – A fast linker for Linux

#38

Post by junon »

ComputerGuru wrote: post #16
senkora wrote: post #9

Earlier quoted context omitted.

I assume that he is referring to "fearless concurrency", the idea that Rust makes it possible to write more complex concurrent programs than other languages because of the safety guarantees: https://doc.rust-lang.org/book/ch16-00-concurrency.html So the logic would go: 1. mold doesn't do incremental linking because it is too complex to do it while still being fast (concurrent). 2. Rust makes it possible to write very…

Actually a lot of the hacks that mold uses to be the fastest linker would be, ironically, harder to reproduce with rust because they’re antithetical to its approach. Eg Mold intentionally eschews used resource collection to speed up execution (it’ll be cleaned up by the os when the process exits) while rust has a strong RAII approach here that would introduce slowdowns.

You can absolutely introduce free-less allocators and the like, as well as use `ManuallyDrop` or `Box::leak`. Rust just asks that you're explicit about it.

Re: Wild – A fast linker for Linux

#39

Post by ignoramous »

devit wrote: post #7

I think the optimal approach for development would be to not produce a traditional linked executable at all, but instead just place the object files in memory, and then produce a loader executable that hooks page faults in those memory areas and on-demand mmaps the relevant object elsewhere, applies relocations to it, and then moves it in place with mremap. Symbols would be resolved based on an index where only updat…

> Symbols would be resolved based on an index where only updated object files are reindexed. It could also eagerly relocate in the background, in order depending on previous usage data.

Not exactly this, but Google's Propeller fixes up ("relinks") Basic Blocks (hot code as traced from PGO) in native code at runtime (like an optimizing JIT compiler would): https://research.google/pubs/propeller-a-profile-guided-reli...

Re: Wild – A fast linker for Linux

#40

Post by sitkack »

ComputerGuru wrote: post #13

There’s been a lot of interest in faster linkers spurred by the adoption and popularity of rust. Even modest statically linked rust binaries can take a couple of minutes in the link stage of compilation in release mode (using mold). It’s not a rust-specific issue but an amalgam of (usually) strictly static linking, advanced link-time optimizations enabled by llvm like LTO and bolt, and a general dissatisfaction with…

I solved this by using Wasm. Your outer application shell calls into Wasm business logic, only the inner logic needs to get recompiled, the outer app shell doesn't even need to restart.
Post reply on HN