Post was not very illuminating. Very little content. It's pretty much a "if musl is slow, it may be the allocator (eom)" which fits in the headline and would have saved me the click.
Why does musl make my Rust code so slow?
61–70 of 77 posts
Re: Why does musl make my Rust code so slow?
#62Earlier quoted context omitted.
Also, it’s extremely common to leave libc dynamically linked for operational reasons. It’s my understanding that by default Go statically links everything but libc.
Go doesn't link libc at all, generally.
Not on Linux, with some exceptions (it wants to play nice with nsswitch, after all). It does/has to on macOS, especially if you want your binaries to work on more than a single system release.
Re: Why does musl make my Rust code so slow?
#63For those curious, Musl's malloc implementation is currently being re-written for higher performance and robustness, see https://github.com/richfelker/mallocng-draft
Curiously, it doesn't adopt the now-standard approach for multithreaded support: per-thread memory pools, allowing one thread allocating and deallocating the same memory to avoid synchronization. This uses one lock guarding allocation, which means that it can be a bottleneck in a multithreaded workload.
Re: Why does musl make my Rust code so slow?
#64Earlier quoted context omitted.
I get that a lot!
Are you familiar with SwiftOnSecurity on twitter? Do you have any hobbies that would be out of character for Intel's Andy Grove? I think the world has room for a ficttionalized Andy Grove talking about how to cook french pastries, train bonsai, intermittent fasting, or preparing for a marathon.
One SwiftOnSecurity is already too many.
Re: Why does musl make my Rust code so slow?
#65Earlier quoted context omitted.
Thanks! That really does seem to be the issue and I wouldn't have known about this, had I not asked. I will try this out and will update the blog post in ~8 hours time.
IME allocations is one of the main things making rust programs slow without diving into the more arcane stuff. So looking into unnecessary allocations and/or the performances of the allocator would be one of the first things to do (right after checking if you're compiling with optimisations). Given your CPU graphs, and the large number of cores, I expect musl's allocator simply has very poor behaviour with respect to…
Tbh allocations (and related memory management things) are often the low hanging fruit in big picture optimization in many languages, incl c++.
Re: Why does musl make my Rust code so slow?
#66Earlier quoted context omitted.
Yeah, upvoted. 30x slower on 48 core (?) system sounds suspiciously like excessive lock contention (or some other shared resource). Non-NUMA aware allocator (or other code) might also contribute to the issue. Should be fairly easy to investigate.
He has so many layers in there it's going to be tough to find the problem. "Ballista is an experimental distributed compute platform, powered by Apache Arrow, with support for Rust and JVM (Java, Kotlin, and Scala)." Plus he's got Docker, the Rust library, musl, and jemalloc sometimes. There's no application. All this is just infrastructure. Musl doesn't do much on its own. But it does do stdio buffering. Could it be…
Excessive I/O is indeed another common issue. The number of times I've seen a slow system and discovered excessive number of flushes, often in something like logging system to be the root cause... Or gazillion of unbuffered 1 byte writes or reads.
Re: Why does musl make my Rust code so slow?
#67Earlier quoted context omitted.
Are you familiar with SwiftOnSecurity on twitter? Do you have any hobbies that would be out of character for Intel's Andy Grove? I think the world has room for a ficttionalized Andy Grove talking about how to cook french pastries, train bonsai, intermittent fasting, or preparing for a marathon.
One SwiftOnSecurity is already too many.
It's a free country. You're allowed to be wrong.
Re: Why does musl make my Rust code so slow?
#68Earlier quoted context omitted.
Go doesn't link libc at all, generally.
Not on Linux, with some exceptions (it wants to play nice with nsswitch, after all). It does/has to on macOS, especially if you want your binaries to work on more than a single system release.
On Linux it has avoided libc for nsswitch in the common case since 2015: https://github.com/golang/go/issues/10485
Golang has only very recently (in its lifetime) and grudgingly admitted that operating systems that provide ABI stability at the DSO layer, rather than the Syscall layer, exist. It has been Linux-first for most of its life, and on Linux syscalls are the ABI stability layer. Not so pretty much anywhere else.
On MacOS, Go links libSystem rather than libc. And this was new in 2018: https://golang.org/doc/go1.11#runtime despite having a Mac port since ~2012. Prior to that they ignored the system ABI stability layer and just did raw Darwin syscalls. They still do so on the BSDs, despite this explicitly not being the supported stability interface.
Re: Why does musl make my Rust code so slow?
#69For those curious, Musl's malloc implementation is currently being re-written for higher performance and robustness, see https://github.com/richfelker/mallocng-draft
Curiously, it doesn't adopt the now-standard approach for multithreaded support: per-thread memory pools, allowing one thread allocating and deallocating the same memory to avoid synchronization. This uses one lock guarding allocation, which means that it can be a bottleneck in a multithreaded workload.
The justifications are partly the same as what Daniel Micay has written extensively on in the rational for hardened_malloc (https://github.com/GrapheneOS/hardened_malloc) - unsynchronized per-thread state inherently sacrifices global consistency for performance and makes it impossible to detect a lot of types of memory usage errors (DF/UAF, etc) that could otherwise be caught.
However musl has the additional constraint of being compatible with small/very-low-memory environments. Lack of global consistency inherently means you will end up using memory less efficiently and requesting significantly more from the system. The new malloc about to go upstream in musl is, to my knowledge, the first/only advanced hardened allocator using slab-type design rather than traditional dlmalloc type split/merge, but also designed for extremely low overhead/waste at low to moderate usage rather than extreme performance. And in the vast majority of applications, this is perfectly reasonable. Even Firefox for example does very well with it.
With that said, new malloc is expected to be somewhat faster than old on lots of workloads (and considerably faster than old would be if we fixed the flaws in old that motivated it), but it's not a performance-oriented allocator. If you really want/need that you should probably link jemalloc or similar (and accept all the tradeoffs that come with that). In Rust programs without "unsafe", it may make sense to do that by default.
Re: Why does musl make my Rust code so slow?
#70We'd be happy to address specific problems on the mailing list. I believe it's a known issue that the Rust compiler is making really heavy use of rapid allocation/freeing cycles, and would benefit from linking a performance-oriented malloc replacement. Doing so is inherently a tradeoff between many factors including performance, memory overhead, safety against erroneous usage by programs, etc.
One statement in your post, which some readers pointed out was apparently added later, "Others have suggested that the performance problems in musl go deeper than that and that there are fundamental issues with threading in musl, potentially making it unsuitable for my use case," seems wrong unless they just meant that the malloc implementation is not thread-caching/thread-local-arena-based. The threads implementation in musl is the only one I'm aware of that doesn't still have significant bugs in some of the synchronization primitives or in cancellation. It's missing a few optional and somewhat obscure features like priority-ceiling mutexes, and Linux doesn't even admit a fully correct implementation in some regards like interaction of thread priorities with some synchronization primitives, but all the basic functionality is there and was written with extreme attention to correctness, and musl aims to be a very good choice in situations where this matters.