Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why does musl make my Rust code so slow?

andygrove.io

Post reply on HN
Print view View original

41–50 of 77 posts

Re: Why does musl make my Rust code so slow?

#41

Post by edwintorok »

jeffbee wrote: post #39
pantalaimon wrote: post #34

Earlier quoted context omitted.

I wonder why glibc isn't just using jemalloc when it seems to perform much better than what it currently has.

Under most workloads jemalloc will use much more memory than ptmalloc (glibc).

I had the opposite experience. Jemalloc uses a lot less (virtual) memory, especially with multithreaded applications. The glibc allocator wastes quite a lot of memory mappings from a thread and never cleans them up even if a thread only had a burst of allocations (there is an upstream bug open about, they don't consider it a leak)

Re: Why does musl make my Rust code so slow?

#42

Post by ashtonkem »

loeg wrote: post #38
ashtonkem wrote: post #20

Earlier quoted context omitted.

Also, it’s extremely common to leave libc dynamically linked for operational reasons. It’s my understanding that by default Go statically links everything but libc.

Go doesn't link libc at all, generally.

I stand corrected.

Re: Why does musl make my Rust code so slow?

#43

Post by edwintorok »

mschuster91 wrote: post #33
MiroF wrote: post #15

Earlier quoted context omitted.

Benchmarking in Docker in general is a mistake I believe.

Why? The only overhead you have in Docker is on syscalls (due to permission checks, namespaces, ...), everything else runs at 100% native speed - unlike assisted virtualization (at least IOMMU overhead plus overhead for anything involving the filesystem) or emulated virtualization (obvious overheads here).

If both sides of the comparison run inside Docker that is still a valid comparison. With Docker the benchmark should be a bit more reproducible by anyone.

Re: Why does musl make my Rust code so slow?

#45

Post by jeffbee »

edwintorok wrote: post #41
jeffbee wrote: post #39

Earlier quoted context omitted.

Under most workloads jemalloc will use much more memory than ptmalloc (glibc).

I had the opposite experience. Jemalloc uses a lot less (virtual) memory, especially with multithreaded applications. The glibc allocator wastes quite a lot of memory mappings from a thread and never cleans them up even if a thread only had a burst of allocations (there is an upstream bug open about, they don't consider it a leak)

I was about to say nobody cares about virtual memory usage, but I can see how those who are extremely sensitive to runtime performance might care about page table entries or VMAs. I think most people are more concerned about RSS.

Re: Why does musl make my Rust code so slow?

#46

Post by liuliu »

jessermeyer wrote: post #29

For those curious, Musl's malloc implementation is currently being re-written for higher performance and robustness, see https://github.com/richfelker/mallocng-draft

Do you have any extra readings on the rationale of building their own malloc rather than integrating mimalloc or jemalloc?

Re: Why does musl make my Rust code so slow?

#47

Post by The_rationalist »

liuliu wrote: post #46
jessermeyer wrote: post #29

For those curious, Musl's malloc implementation is currently being re-written for higher performance and robustness, see https://github.com/richfelker/mallocng-draft

Do you have any extra readings on the rationale of building their own malloc rather than integrating mimalloc or jemalloc?

The NIH syndrome obviously, otherwise it would have been addressed at the start of the Readme

Re: Why does musl make my Rust code so slow?

#48

Post by jessermeyer »

liuliu wrote: post #46
jessermeyer wrote: post #29

For those curious, Musl's malloc implementation is currently being re-written for higher performance and robustness, see https://github.com/richfelker/mallocng-draft

Do you have any extra readings on the rationale of building their own malloc rather than integrating mimalloc or jemalloc?

Not any first hand, but reading their principles suggests that simplicity and ease of deployment are probably relevant. https://musl.libc.org/about.html

Re: Why does musl make my Rust code so slow?

#49

Post by harrygeez »

liuliu wrote: post #46
jessermeyer wrote: post #29

For those curious, Musl's malloc implementation is currently being re-written for higher performance and robustness, see https://github.com/richfelker/mallocng-draft

Do you have any extra readings on the rationale of building their own malloc rather than integrating mimalloc or jemalloc?

well one of the stated goals of musl is to be simple and correct, and all those mallocs are anything but simple

Re: Why does musl make my Rust code so slow?

#50

Post by asveikau »

aidenn0 wrote: post #35
tyingq wrote: post #26

Earlier quoted context omitted.

Ah, okay. Searched a bit, and it apparently requires you to find your own with dns client lookup library, avoid dlopen(), set GCONV_PATH, and so on.

For a long time glibc was maintained by someone who was of the opinion that if you wanted to statically link 3rd party libraries, then you shouldn't be allowed to build binaries.

It's interesting because there is an opposite cult (I believe some prominent plan9 and go people are adherents) that believes dynamic linking should literally not be a thing. I even recall reading some essays that introduction of a dynamic linker was some kind of tragic downfall in Unix history.

I think the truth is that both have costs and benefits. Dynamic linking is good for security patches, memory and disk usage. But it creates new opportunities for problems, for example ABI breakage becomes more significant of a problem and needs a lot of care to avoid. People distributing code, be it to end users or app stores or on servers with things like chroots, jails and containers, need to carry their dependencies anyway negating some of the benefits.

Post reply on HN