Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Spotting and avoiding heap fragmentation in Rust applications

svix.com

Post reply on HN
Print view View original

41–50 of 70 posts

Re: Spotting and avoiding heap fragmentation in Rust applications

#41

Post by sgeisenh »

munificent wrote: post #25

In some ways Rust is in the worst possible position in terms of language design when it comes to fragmentation. In a completely manually managed language like C or C++, you can handle fragmentation problems yourself by writing your own allocators or doing object pools to reuse previously allocated memory. You have control over fragmentation. In a garbage collected language like Java or C#, the runtime is able to move…

Doesn't the normal approach in C++ run into similar issues? The workarounds in C++ are more ergonomic than in Rust, but can still require a lot of refactoring.

One of the performance bottlenecks that I ran into while reimplementing clox in C++ was using `new` and `delete` instead of `realloc` for arrays and hash tables. By the time that I figured out it was an issue, I was already using `operator new` to track heap allocations for the garbage collector.

Re: Spotting and avoiding heap fragmentation in Rust applications

#42

Post by tayo42 »

tasn wrote: post #40
tayo42 wrote: post #39

was curious about > When services exit abruptly, this can lower availability... which is bad for business. interesting company > Do you offer SLAs? We offer uptime SLAs of 99.999% for our enterprise customers, 99.99% for our business tier customers, and 99.9% for our startup tier users. 99.999 is miserable to support. how do you even get the granularity to measure, freedom to try anything? i dont think cloud services…

> 99.999 is miserable to support. how do you even get the granularity to measure, freedom to try anything? i dont think cloud services even provide slas like that for their services. A lot of redundancies and testing. > 1 billion a year would average out to 32 a second. if its 10 billion? 320/second. surprising to see rust for this. gc languages should be able to handle it. This assumes even distribution, though traf…

Oh maybe I missed but I didn't think that info was there. Would have been interesting to see thr extent of fragmentation or what the cause was.

5 9s of uptime means 800ms of down time a day. Is this not in a cloud, a load balancer health check can't react that fast. Vms will get randomly killed if something is wrong.

Re: Spotting and avoiding heap fragmentation in Rust applications

#43

Post by tasn »

tayo42 wrote: post #42
tasn wrote: post #40

Earlier quoted context omitted.

> 99.999 is miserable to support. how do you even get the granularity to measure, freedom to try anything? i dont think cloud services even provide slas like that for their services. A lot of redundancies and testing. > 1 billion a year would average out to 32 a second. if its 10 billion? 320/second. surprising to see rust for this. gc languages should be able to handle it. This assumes even distribution, though traf…

Oh maybe I missed but I didn't think that info was there. Would have been interesting to see thr extent of fragmentation or what the cause was. 5 9s of uptime means 800ms of down time a day. Is this not in a cloud, a load balancer health check can't react that fast. Vms will get randomly killed if something is wrong.

The SLA is monthly, not daily.

Re: Spotting and avoiding heap fragmentation in Rust applications

#44

Post by robot_no_421 »

munificent wrote: post #25

In some ways Rust is in the worst possible position in terms of language design when it comes to fragmentation. In a completely manually managed language like C or C++, you can handle fragmentation problems yourself by writing your own allocators or doing object pools to reuse previously allocated memory. You have control over fragmentation. In a garbage collected language like Java or C#, the runtime is able to move…

It's not really that hard of a problem. You just use a different allocator (like the author did in this article). In fact, you can even write your own allocator with unsafe if you want (jemalloc is already using a lot of unsafe in the code anyway, you need to use it to build any real allocator).

'unsafe' doesn't equal 'don't use'. I'll admit that unsafe Rust has a lot of rough and unergonomic features compared to C, but it's conceptually and algorithmically just as easy (or hard) in Rust to build an allocator as it is in C and C++.

You also still get the help of the borrow checker in unsafe rust, so there are even some advantages.

Re: Spotting and avoiding heap fragmentation in Rust applications

#45

Post by jeffbee »

scottlamb wrote: post #31
mcronce wrote: post #13

Earlier quoted context omitted.

I agree that there could have been a more satisfying conclusion, but it is worth noting that jemalloc isn't a panacea. I've seen issues similar to Svix in both Rust and C++ applications that were heavy on ephemeral allocations, and have fixed it by doing all of the following, depending on the specific process: * Switching from libc malloc to jemalloc * Switching from libc malloc to tcmalloc (dating myself a little bi…

> * Switching from libc malloc to tcmalloc (dating myself a little bit) If you think of tcmalloc as an old crusty allocator, you've probably only seen the gperftools version of it. This is the version Google now uses internally: https://github.com/google/tcmalloc It's worth a fresh look. In particular, it supports per-CPU caches as an alternative to per-thread caches. Those are fantastic if you have a lot more thread…

I think we can safely say that jemalloc has much better marketing than tcmalloc. If you search on github there is only 1 real project with a bazel WORKSPACE that mentions tcmalloc ... the rest are either forks of tcmalloc itself or trivial toy programs that I personally put on github to share with someone. There are zillions of users of jemalloc and everyone has heard of it.

Re: Spotting and avoiding heap fragmentation in Rust applications

#46

Post by glandium »

MuffinFlavored wrote: post #2

This article would've been a bit cooler if the conclusion wasn't "switch from default allocator to jemalloc" but instead "use jemalloc to prove something is wrong in the default allocator and track down + find a fix for what's wrong in the default allocator" Unless I misunderstood that the default Rust allocator, with high request bodies and concurrency, is always going to suffer unfixable heap fragmentation like dis…

Reading the article reminded me of https://sourceware.org/bugzilla/show_bug.cgi?id=23416 and I wouldn't be surprised if it's the same underlying issue.

Re: Spotting and avoiding heap fragmentation in Rust applications

#47

Post by scottlamb »

jeffbee wrote: post #45
scottlamb wrote: post #31

Earlier quoted context omitted.

> * Switching from libc malloc to tcmalloc (dating myself a little bit) If you think of tcmalloc as an old crusty allocator, you've probably only seen the gperftools version of it. This is the version Google now uses internally: https://github.com/google/tcmalloc It's worth a fresh look. In particular, it supports per-CPU caches as an alternative to per-thread caches. Those are fantastic if you have a lot more thread…

I think we can safely say that jemalloc has much better marketing than tcmalloc. If you search on github there is only 1 real project with a bazel WORKSPACE that mentions tcmalloc ... the rest are either forks of tcmalloc itself or trivial toy programs that I personally put on github to share with someone. There are zillions of users of jemalloc and everyone has heard of it.

Agreed. I think it's a wider phenomenon: Google is culturally uninterested in / incapable of marketing an open source project that's less ambitious / strategic than e.g. TensorFlow, Kubernetes, Go, Dart/Flutter, or (looking back a few years...) Angular. The next tier I guess is Abseil or Bazel, which have a nice website, nice docs, and some papers/talks but I think still aren't super widely used outside Google. I can't think of anything smaller than those that's gotten any marketing at all. Can you?

tcmalloc at least gets internal changes regularly synced to github (since just a couple years ago iirc), vs. the ancient gperftools snapshot that's more widely known. And there are many other projects of potential interest to the outside world (Fibers...) that have been mentioned publicly but not open sourced at all.

Re: Spotting and avoiding heap fragmentation in Rust applications

#48

Post by vlovich123 »

jeffbee wrote: post #45
scottlamb wrote: post #31

Earlier quoted context omitted.

> * Switching from libc malloc to tcmalloc (dating myself a little bit) If you think of tcmalloc as an old crusty allocator, you've probably only seen the gperftools version of it. This is the version Google now uses internally: https://github.com/google/tcmalloc It's worth a fresh look. In particular, it supports per-CPU caches as an alternative to per-thread caches. Those are fantastic if you have a lot more thread…

I think we can safely say that jemalloc has much better marketing than tcmalloc. If you search on github there is only 1 real project with a bazel WORKSPACE that mentions tcmalloc ... the rest are either forks of tcmalloc itself or trivial toy programs that I personally put on github to share with someone. There are zillions of users of jemalloc and everyone has heard of it.

We use it here at Cloudflare on every single machine as part of Workers. So that’s two major hyperscalers running large RAM multi tenant workloads.

Jemalloc may more recognition in the broader community, but the largest workloads seem to be running mimalloc / tcmalloc (I don’t know what Facebook uses internally). The libc malloc probably has even more users than either as it’s the default allocator for iOS and Android.

Re: Spotting and avoiding heap fragmentation in Rust applications

#49

Post by vlovich123 »

Asooka wrote: post #33
adrianmonk wrote: post #28

> Gaps that are too small and scattered throughout the heap can lead to new "fresh" blocks of memory being allocated to accommodate a new value that won’t fit otherwise. Though unfortunately because of how memory management works a "defrag" is not possible. This is how memory management works now, but some older systems like classic Mac OS and Palm OS used a design that did make it possible to compact the heap. See h…

Rust's borrow checker should make it impossible to reference an unlocked pointer in safe code. I suspect it should be possible to have a compacting heap as a library.

Yeah. I wonder if something like this might be interesting for a pure Rust program / what computing would look like if Rust were available back then (we now have a path dependency on modern memory allocation patterns that are unlikely to change where we rely on the allocator to do really heavy lifting). I wonder how expensive the extra handle indirection would be (I imagine modern CPUs should be an able to speculate through it without much difficulty).

Re: Spotting and avoiding heap fragmentation in Rust applications

#50

Post by josephg »

munificent wrote: post #25

In some ways Rust is in the worst possible position in terms of language design when it comes to fragmentation. In a completely manually managed language like C or C++, you can handle fragmentation problems yourself by writing your own allocators or doing object pools to reuse previously allocated memory. You have control over fragmentation. In a garbage collected language like Java or C#, the runtime is able to move…

I hear what you're saying, and I think I agree - I just frame it slightly differently. As others have mentioned, you can use custom allocators & object pooling in rust. I'm a big fan of bumpalo.

But its awkward. I think this awkwardness comes because rust is trying to straddle the gap between being a low level systems language and a high level language for application developers.

In a systems language, I want full control over the allocator for each allocation my program does. Do I want to use the system allocator, or an arena or an object pool? Different tasks call for different tools. I really like Zig's approach here where every collection type which allocates takes an allocator as a parameter when the object is created. Rust's borrow checker can be super helpful here in making all of this stuff correct.

But application developers don't want to think about allocators and memory fragmentation. Adding allocator traits to everything will add yet more things to learn in rust. And rust's standard library doesn't support that (yet).

The result is that allocation crates like bumpalo need to ship with & maintain their own copy of the standard library's collection types. Its kind of a mess. This might eventually be fixed, but at the rate rust's development has been going lately, I suspect it'll take years for something to land on stable.

But people in the community are talking about it, at least. Eg this article from the rust subreddit: https://nical.github.io/posts/rust-custom-allocators.html

Post reply on HN