run a perf trace on both and see what jumps out
If he's benchmarking on docker, I'm not sure that perf works in docker.
Depends on the kernel.perf_event_paranoid sysctl.
Why does musl make my Rust code so slow?
31–40 of 77 posts
Re: Why does musl make my Rust code so slow?
#32This actually someone asking and not an investigation and explanation. There isn't even a lot of due diligence to figuring it out - no profiling or resource usage other than CPUs. Also it is musl combined with docker causing a 30x slowdown. If something is running 30x slower from linking in a different libc, I'm guessing it should not be that difficult to narrow down the cause at least a little bit.
Exactly, not sure why you're downvoted. His blog-post doesn't explain why it is slow but instead 'switches the allocator' and says he has "fixed" the issue. This is just a workaround, not a fix.
I'm quite surprised that there's no mention of profiling the actual allocator causing this regression to properly narrow the fault down to the source. Instead this blog-post encourages cargo-culting development to "fix slow code".
Re: Why does musl make my Rust code so slow?
#33This actually someone asking and not an investigation and explanation. There isn't even a lot of due diligence to figuring it out - no profiling or resource usage other than CPUs. Also it is musl combined with docker causing a 30x slowdown. If something is running 30x slower from linking in a different libc, I'm guessing it should not be that difficult to narrow down the cause at least a little bit.
Benchmarking in Docker in general is a mistake I believe.
Why? The only overhead you have in Docker is on syscalls (due to permission checks, namespaces, ...), everything else runs at 100% native speed - unlike assisted virtualization (at least IOMMU overhead plus overhead for anything involving the filesystem) or emulated virtualization (obvious overheads here).
Re: Why does musl make my Rust code so slow?
#34Earlier quoted context omitted.
Without familiarity with rust, I wasn't sure what they meant by "system allocator". Apparently that means libc's malloc. (Or HeapAlloc on Windows) So I guess they statically link jemalloc but can optionally use libc malloc.
It's the other way around now, though the post that iou linked is from an earlier period when things worked differently. By default today, Rust programs use the same allocator that C programs use, which I think is provided by libc on Linux, and you have the option of using a custom allocator, like jemalloc. Historically however, all Rust programs used to use jemalloc by default.
I wonder why glibc isn't just using jemalloc when it seems to perform much better than what it currently has.
Re: Why does musl make my Rust code so slow?
#35Earlier quoted context omitted.
I think it's mainly that glibc has poor support for statically linking.
Ah, okay. Searched a bit, and it apparently requires you to find your own with dns client lookup library, avoid dlopen(), set GCONV_PATH, and so on.
For a long time glibc was maintained by someone who was of the opinion that if you wanted to statically link 3rd party libraries, then you shouldn't be allowed to build binaries.
Re: Why does musl make my Rust code so slow?
#36Swap out the allocator https://users.rust-lang.org/t/optimizing-rust-binaries-obser...
Thanks! That really does seem to be the issue and I wouldn't have known about this, had I not asked. I will try this out and will update the blog post in ~8 hours time.
IME allocations is one of the main things making rust programs slow without diving into the more arcane stuff. So looking into unnecessary allocations and/or the performances of the allocator would be one of the first things to do (right after checking if you're compiling with optimisations).
Given your CPU graphs, and the large number of cores, I expect musl's allocator simply has very poor behaviour with respect to multithreading (e.g. limited or no threadlocal arenas, size-classing, etc…) leading to a lot of crosstalk, extreme contention on allocations, etc...
Re: Why does musl make my Rust code so slow?
#37Earlier quoted context omitted.
I think it's mainly that glibc has poor support for statically linking.
Ah, okay. Searched a bit, and it apparently requires you to find your own with dns client lookup library, avoid dlopen(), set GCONV_PATH, and so on.
In general, do not expect dlopen() to work in statically linked programs (in any libc).
Re: Why does musl make my Rust code so slow?
#38Earlier quoted context omitted.
I suppose there's no option to link statically against glibc because of the implications with LGPL (static linking triggers additional license clauses).
Also, it’s extremely common to leave libc dynamically linked for operational reasons. It’s my understanding that by default Go statically links everything but libc.
Go doesn't link libc at all, generally.
Re: Why does musl make my Rust code so slow?
#39Earlier quoted context omitted.
It's the other way around now, though the post that iou linked is from an earlier period when things worked differently. By default today, Rust programs use the same allocator that C programs use, which I think is provided by libc on Linux, and you have the option of using a custom allocator, like jemalloc. Historically however, all Rust programs used to use jemalloc by default.
I wonder why glibc isn't just using jemalloc when it seems to perform much better than what it currently has.
Under most workloads jemalloc will use much more memory than ptmalloc (glibc).
Re: Why does musl make my Rust code so slow?
#40Earlier quoted context omitted.
Sure. Wasn't suggesting it as the default. Just curious why it wasn't an option.
I always assumed that that’s done for portability reasons, since libc deals with a lot of os specific details.
Symbol versioning for backwards compatibility is only possible with dynamic libraries, for example. Static linking is of course always backwards compatible for a given binary artifact, but is significantly bloatier and makes it harder to apply security fixes.