Can it link the Linux kernel yet? Was a useful milestone for LLD.
Not yet. The Linux kernel uses linker scripts, which Wild doesn't yet support. I'd like to add support for linker scripts at some point, but it's some way down the priority list.
Wild – A fast linker for Linux
121–130 of 222 posts
Re: Wild – A fast linker for Linux
#122What would be refreshing would be a C/C++ compiler that did away with the intermediate step of linking and built the whole program as a unit. LTO doesn't even have to be a thing if the compiler can see the entire program in the first place. It would still have to save some build products so that incremental builds are possible, but not as object files, the compiler would need metadata to know of the origin and depend…
SQLite3 just concatenation everything together into one compilation unit. So, more people have been using this than probably know about it. https://sqlite.org/amalgamation.html
*concatenates
Apologies for the typo. And now it is too late to edit the post.
Re: Wild – A fast linker for Linux
#123Is it too late to ask what a linker is?
Re: Wild – A fast linker for Linux
#124I'm curious: what's the theory behind why this would be faster than mold in the non-incremental case? "Because Rust" is a fine explanation for a bunch of things, but doesn't explain expected performance benefits. "Because there's low hanging concurrent fruit that Rust can help us get?" would be interesting but that's not explicitly stated or even implied.
I'm not actually sure, mostly because I'm not really familiar with the Mold codebase. One clue is that I've heard that Mold gets about a 10% speedup by using a faster allocator (mimalloc). I've tried using mimalloc with Wild and didn't get any measurable speedup. This suggests to me that Mold is probably making heavier use of the allocator than Wild is. With Wild, I've certainly tried to optimise the number of heap allocations.
But in general, I'd guess just different design decisions. As for how this might be related to Rust - I'm certain that were Wild ported from Rust to C or C++, that it would perform very similarly. However, code patterns that are fine in Rust due to the borrow checker, would be footguns in languages like C or C++, so maintaining that code could be tricky. Certainly when I've coded in C++ in the past, I've found myself coding more defensively, even at a small performance cost, whereas with Rust, I'm able to be a lot bolder because I know the compiler has got my back.
Re: Wild – A fast linker for Linux
#125Earlier quoted context omitted.
> Traditional allocators are fast Really it's allocators in general. Allocations are perceived as expensive only because they are mostly dependent on the amortized cost of prior deallocations. As an extreme example, even GCs can be fast if you avoid deallocation because most typically have a long-lived object heap that rarely gets collected - so if you keep things around that can be long-lived (pooling) their cost mo…
Slight disagreement here. Allocation is perceived as slow because it is. Getting memory from the OS is somewhat expensive because a page of memory needs to be allocated and stored off. Getting memory from traditional allocators is expensive because freespace needs to be tracked. When you say "I need 5 bytes" the allocator needs to find 5 free bytes to give back to you. Bump allocators are fast because the operation o…
> Getting memory from traditional allocators is expensive because freespace needs to be tracked.
If you've never deallocated then there is no free space to track, hence the cost of allocation is _mostly_ (per my above comment) affected by the amortized cost of deallocations. The cost becomes extremely apparent with GCs because a failed allocation is usually what triggers a collection (and subsequently any deallocations that need to happen).
Still, your comment goes into detail that I probably should have.
Re: Wild – A fast linker for Linux
#126Earlier quoted context omitted.
Isn't this how dynamic linking works? If you really want to reduce build times, you should be making your hot path in the build a shared library, so you don't have to relink so long as you're not changing the interface.
But do rust’s invariants work across dynamic links? I thought a lot of its proofs were done at compile time not link time.
Yesn't.
Rust is perfectly happy to emit/use dynamic links.[0] It's just that the primary C use case (distributing and updating the main app and its libraries separately) ends up being unsafe since Rust's ABI is unstable (so compiler versions, libraries, etc must match exactly).
Avoiding static relinking during development is pretty much the use where it does work. In fact, Bevy recommends this as part of its setup guide![1]
Practice paints a slightly less rosy picture, though; since the feature is exercised quite rarely, not all libraries work well with it in practice.[2]
[0]: https://doc.rust-lang.org/reference/linkage.html#r-link.dyli...
[1]: https://bevyengine.org/learn/quick-start/getting-started/set...
[2]: For example, https://github.com/linebender/bevy_vello/issues/84
Re: Wild – A fast linker for Linux
#127Is it too late to ask what a linker is?
I'll ELI5:
Compilers take the code the programmer writes, and turns it into things called object files. Object files are close to executable by the target processor, but not completely. There are little places where the code needs to be rewritten to handle access to subroutines, access operating system functionality, and other things.
A linker combines all these object files, does the necessary rewriting, and generates something that the operating system can use.
It's the final step in building an executable.
--
More complicatedly: a linker is a little Turing machine that runs over the object files. Some can do complicated things like rewriting code, or optimizing across function calls. But, fundamentally, they plop all the object files together and follow little scripts (or rewrites) that clean up the places the compiler couldn't properly insert instructions because the compiler doesn't know the final layout of the program.
Re: Wild – A fast linker for Linux
#128Ever since mold relicensed from AGPL to MIT (as part of mold 2.0 release), the worldwide need for making another fast linker has been greatly reduced, so I wasn't expecting a project like this to appear. And definitely wasn't expecting it to already be 2x faster than mold in some cases. Will keep an eye on this project to see how it evolves, best of luck to the author.
Note that Mold has no interest in becoming incremental, so there is a big reason there for another linker to exist. I find it kind of embarrassing that MS' linker has been incremental by default for decades, yet there's no production ready incremental linker on Linux yet.
OTOH even lld, fast but fairly slower than mold, is already incredibly faster than MS's linker even without the incrmeentality. Like, I'm routinely linking hundreds of megabytes in less than a second anyways, not sure incrementality is that much worth it
Re: Wild – A fast linker for Linux
#129I looked at this before, is it ready for production? I thought not based on the readme, so I'm still using mold. For those on macOS, Apple released a new linker about a year or two ago (which is why the mold author stopped working on their macOS version), and if you're using it with Rust, put this in your config.toml: [target.aarch64-apple-darwin] rustflags = [ "-C", "link-arg=-fuse-ld=/Applications/Xcode.app/Content…
Isn't the new linked just the default these days? I'm not sure adding that has any effect.
Re: Wild – A fast linker for Linux
#130Earlier quoted context omitted.
Note that Mold has no interest in becoming incremental, so there is a big reason there for another linker to exist. I find it kind of embarrassing that MS' linker has been incremental by default for decades, yet there's no production ready incremental linker on Linux yet.
OTOH even lld, fast but fairly slower than mold, is already incredibly faster than MS's linker even without the incrmeentality. Like, I'm routinely linking hundreds of megabytes in less than a second anyways, not sure incrementality is that much worth it
Not a rhetorical question: Could it be that part of the speed difference is due to the file system speed? I was shocked when I saw how much modern(ish) Windows file systems were slower than modern(ish) Linux ones.