Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Wild – A fast linker for Linux

github.com

Post reply on HN
Print view View original

41–50 of 222 posts

Re: Wild – A fast linker for Linux

#41

Post by the_duke »

throwaway106382 wrote: post #4

> Mold is already very fast, however it doesn't do incremental linking and the author has stated that they don't intend to. Wild doesn't do incremental linking yet, but that is the end-goal. By writing Wild in Rust, it's hoped that the complexity of incremental linking will be achievable. Can someone explain what is so special about Rust for this?

Rust has a pretty good incremental caching compiler that makes debug builds relatively fast.

Linking is often a very notable bottleneck for debug binaries and mold can make a big difference.

So interest in speeding up linking for Rust is expected.

Re: Wild – A fast linker for Linux

#42

Post by ComputerGuru »

sitkack wrote: post #40
ComputerGuru wrote: post #13

There’s been a lot of interest in faster linkers spurred by the adoption and popularity of rust. Even modest statically linked rust binaries can take a couple of minutes in the link stage of compilation in release mode (using mold). It’s not a rust-specific issue but an amalgam of (usually) strictly static linking, advanced link-time optimizations enabled by llvm like LTO and bolt, and a general dissatisfaction with…

I solved this by using Wasm. Your outer application shell calls into Wasm business logic, only the inner logic needs to get recompiled, the outer app shell doesn't even need to restart.

I don’t think I can use wasm with simd or syscalls, which is the bulk of my work.

Re: Wild – A fast linker for Linux

#43

Post by zamalek »

cma wrote: post #35
cogman10 wrote: post #25

Earlier quoted context omitted.

Depends on how things are approached. You could, for example, take advantage of bump arena allocator in rust which would allow the linker to have just 1 alloc/dealloc. Mold is still using more traditional allocators under the covers which won't be as fast as a bump allocator. (Nothing would stop mold from doing the same).

Traditional allocators are fast if you never introduce much fragmentation with free, though you may still get some gaps and have some other overhead and not be quite as fast. But why couldn't you just LD_PRELOAD a malloc for mold that worked as a bump/stack/arena allocator and just ignored free if anything party stuff isn't making that many allocations?

> Traditional allocators are fast

Really it's allocators in general. Allocations are perceived as expensive only because they are mostly dependent on the amortized cost of prior deallocations. As an extreme example, even GCs can be fast if you avoid deallocation because most typically have a long-lived object heap that rarely gets collected - so if you keep things around that can be long-lived (pooling) their cost mostly goes away.

Re: Wild – A fast linker for Linux

#44

Post by estebank »

pzmarzly wrote: post #19

Ever since mold relicensed from AGPL to MIT (as part of mold 2.0 release), the worldwide need for making another fast linker has been greatly reduced, so I wasn't expecting a project like this to appear. And definitely wasn't expecting it to already be 2x faster than mold in some cases. Will keep an eye on this project to see how it evolves, best of luck to the author.

Note that Mold has no interest in becoming incremental, so there is a big reason there for another linker to exist. I find it kind of embarrassing that MS' linker has been incremental by default for decades, yet there's no production ready incremental linker on Linux yet.

Re: Wild – A fast linker for Linux

#45

Post by dralley »

satvikpendem wrote: post #12

I looked at this before, is it ready for production? I thought not based on the readme, so I'm still using mold. For those on macOS, Apple released a new linker about a year or two ago (which is why the mold author stopped working on their macOS version), and if you're using it with Rust, put this in your config.toml: [target.aarch64-apple-darwin] rustflags = [ "-C", "link-arg=-fuse-ld=/Applications/Xcode.app/Content…

No, the author is pretty clear that it shouldn't be used for production yet

Re: Wild – A fast linker for Linux

#46

Post by 95014_refugee »

devit wrote: post #7

I think the optimal approach for development would be to not produce a traditional linked executable at all, but instead just place the object files in memory, and then produce a loader executable that hooks page faults in those memory areas and on-demand mmaps the relevant object elsewhere, applies relocations to it, and then moves it in place with mremap. Symbols would be resolved based on an index where only updat…

This makes some very naïve assumptions about the relationships between entities in a program; in particular that you can make arbitrary assertions about the representation of already-allocated datastructures across multiple versions of a component, that the program's compositional structure morphs in understandable ways, and that you can pause a program in a state where a component can actually be replaced.

By the time you have addressed these, you'll find yourself building a microkernel system with a collection of independent servers and well-defined interaction protocols. Which isn't necessarily a terrible way to assemble something, but it's not quite where you're trying to go...

Re: Wild – A fast linker for Linux

#47

Post by dralley »

ComputerGuru wrote: post #16
senkora wrote: post #9

Earlier quoted context omitted.

I assume that he is referring to "fearless concurrency", the idea that Rust makes it possible to write more complex concurrent programs than other languages because of the safety guarantees: https://doc.rust-lang.org/book/ch16-00-concurrency.html So the logic would go: 1. mold doesn't do incremental linking because it is too complex to do it while still being fast (concurrent). 2. Rust makes it possible to write very…

Actually a lot of the hacks that mold uses to be the fastest linker would be, ironically, harder to reproduce with rust because they’re antithetical to its approach. Eg Mold intentionally eschews used resource collection to speed up execution (it’ll be cleaned up by the os when the process exits) while rust has a strong RAII approach here that would introduce slowdowns.

Nothing about Rust requires the use of the heap or RAII.

Also, if wild is indeed faster than mold even without incrementalism, as the benchmarks show, then it seems quite silly to go around making the argument that it's harder to write a fast linker in Rust. It's apparently not that hard.

Re: Wild – A fast linker for Linux

#48

Post by jjmarr »

devit wrote: post #7

I think the optimal approach for development would be to not produce a traditional linked executable at all, but instead just place the object files in memory, and then produce a loader executable that hooks page faults in those memory areas and on-demand mmaps the relevant object elsewhere, applies relocations to it, and then moves it in place with mremap. Symbols would be resolved based on an index where only updat…

Isn't this how dynamic linking works? If you really want to reduce build times, you should be making your hot path in the build a shared library, so you don't have to relink so long as you're not changing the interface.

Re: Wild – A fast linker for Linux

#49

Post by tialaramex »

compiler-guy wrote: post #14
throwaway106382 wrote: post #4

> Mold is already very fast, however it doesn't do incremental linking and the author has stated that they don't intend to. Wild doesn't do incremental linking yet, but that is the end-goal. By writing Wild in Rust, it's hoped that the complexity of incremental linking will be achievable. Can someone explain what is so special about Rust for this?

That’s puzzling to me too. Rust is a great language, and probably makes developing Wild faster. But the complexity of incremental linking doesn’t stem from the linker’s implementation language. It stems from all the tracking, reserved spacing, and other issues required to link a previously linked binary (or at least parts of it) a second time.

I would guess the idea is that in Rust the complexity is cheaper on a "per unit" basis so you can afford more complexity. So yes, it is a more complicated problem than the previous linkers, but, in Rust maybe you can get that done anyway.

Re: Wild – A fast linker for Linux

#50

Post by kryptiskt »

What would be refreshing would be a C/C++ compiler that did away with the intermediate step of linking and built the whole program as a unit. LTO doesn't even have to be a thing if the compiler can see the entire program in the first place. It would still have to save some build products so that incremental builds are possible, but not as object files, the compiler would need metadata to know of the origin and dependencies of all the generated code so it would be able to replace the right things.

External libs are most often linked dynamically these days, so they don't need to be built from source, so eliminating the linker doesn't pose a problem for non-open source dependencies. And if that's not enough letting the compiler also consume object files could provide for legacy use cases or edge cases where you must statically link to a binary.

Post reply on HN