[dead]
Meta’s renewed commitment to jemalloc
231–240 of 259 posts
Re: Meta’s renewed commitment to jemalloc
#232Earlier quoted context omitted.
Well, let me just circle back to the start of this comment chain. > Many programs in GC language end up fighting the GC by allocating a large buffer and managing it by hand That's the primary thing I'm contending with. This is a strategy for fighting the GC, but it's also generally a bad strategy. One that I think gets pulled more because someone heard of the suggestion and less because it's a good way to make things…
This is a strategy for fighting the GC, but it's also generally a bad strategy. Allocating a large buffer is literally what an array or vector is. A heap uses a heap structure and hops around in memory for every allocation and free. It gets worse the more allocations there are. The allocations are fragmented and in different parts of memory. Allocating a large buffer takes care of all this if it is possible to anythi…
> A heap uses a heap structure and hops around in memory for every allocation and free.
Not in the JVM. And maybe this is ultimately what we are butting up against. After all, the JVM isn't all GCed languages, it's just one of many.
In the JVM, heap allocations are done via bump allocation. When a region is filled, the JVM performs a garbage collection which moves objects in the heap (it compacts the memory). It's not an actual heap structure for the JVM.
> It doesn't make sense to make lots of heap allocations when what you want is multiple items next to each other in memory and one heap allocation.
That is (currently) not possible to do in the JVM, barring primitives. When I create a `new Foo[128]` in the JVM, that creates an array big enough to hold 128 references of Foo, not 128 Foo objects. Those have to be allocated onto the heap separately. This is part of the reason why managing such an object pool is pointless in the JVM. You have to make the allocations anyways and you are paying for the management cost of that pool.
The object pool is also particularly bad in the JVM because it stops the JVM from performing optimizations like scalarization. That's where the JVM can completely avoid a heap allocation all together and instead pulls out the internal fields of the allocated object to hand off to a calling function. In order for that optimization to occur, and object can't escape the current scope.
I get why this isn't the same story if you are talking about another language like C# or go. There are still the negative consequences of needing to manage the buffer, especially if the intent is to track allocations of items in the buffer and to reassign them. But there is a gain in the locality that's nice.
> Using indices of an array is not difficult and neither is handing out indices or ranges to in small sections.
Easy to do? Sure. Easy to do fast? Well, no. That's entirely the reason why C++ has multiple allocators. It's the crux of the problem an allocator is trying to solve in the first place "How can I efficiently give a chunk of memory back to the application".
Obviously, it'll matter what your usage pattern is, but if it's at all complex, you'll run into the same problems that the general allocator hits.
Re: Meta’s renewed commitment to jemalloc
#233> With the leverage jemalloc provides however, it can be tempting to realize some short-term benefit. It requires strong self-discipline as an organization to resist that temptation and adhere to the core engineering principles. This doesn't quite read properly to me. What does it actually mean, does anyone know?
I'm pretty sure it means something like this: "Because jemalloc is used all over the place in our systems that run at tremendous scale, some hack that improves its performance a little bit while degrading the longer-term maintainability of the code can look very appealing -- look, doing this thing will save us $X,000,000 per year! -- and it takes discipline to avoid giving in to that temptation and to insist on doing things properly even if sometimes it means passing up a chance to make the code 0.1% faster and 10% messier."
Re: Meta’s renewed commitment to jemalloc
#234[dead]
Re: Meta’s renewed commitment to jemalloc
#235Earlier quoted context omitted.
This is a strategy for fighting the GC, but it's also generally a bad strategy. Allocating a large buffer is literally what an array or vector is. A heap uses a heap structure and hops around in memory for every allocation and free. It gets worse the more allocations there are. The allocations are fragmented and in different parts of memory. Allocating a large buffer takes care of all this if it is possible to anythi…
> A heap uses a heap structure and hops around in memory for every allocation and free. Not in the JVM. And maybe this is ultimately what we are butting up against. After all, the JVM isn't all GCed languages, it's just one of many. In the JVM, heap allocations are done via bump allocation. When a region is filled, the JVM performs a garbage collection which moves objects in the heap (it compacts the memory). It's no…
In the JVM, heap allocations are done via bump allocation.
If that were true then they wouldn't be heap allocations.
https://www.digitalocean.com/community/tutorials/java-jvm-me...
https://docs.oracle.com/en/java/javase/21/core/heap-and-heap...
not possible to do in the JVM, barring primitives
Then you make data structures out of arrays of primitives.
Easy to do? Sure. Easy to do fast? Well, no. That's entirely the reason why C++ has multiple allocators.
I don't know what this means. Vectors are trivial and if you hand out ranges of memory in an arena allocator you allocate it once and free it once which solves the heavy allocation problem. The allocator parameter in templates don't factor in to this.
Re: Meta’s renewed commitment to jemalloc
#236Earlier quoted context omitted.
You really need to benchmark your workloads, ideally with the "big 3" (jemalloc, tcmalloc, mimalloc). They all have their strengths and weaknesses. Jemalloc can usually keep the smallest memory footprint, followed by tcmalloc. Mimalloc can really speed things up sometimes. As usually, YMMV.
Look up the numbers in other comments above. When it comes to performance, the Google's tcmalloc is unconquered.
I tried all three, multiple times, and it depends.
Using the last workload tested as an example, mimalloc just consumed memory like crazy. It was probably leaking, as it was the stock version that comes in Debian, so probably quite old.
Tcmalloc and jemalloc were neck to neck when comparing app metrics (request duration etc... was quite similar), but jemalloc consistently used only about half of RAM as opposed to tcmalloc).
Both custom allocators used way less RAM than the stock allocator though. Something like 10x (!) less. In the end the workload with jemalloc hovers somewhere around 4% of the memory limit. Not bad for one single package and an additional compile option to enable it.
Re: Meta’s renewed commitment to jemalloc
#237Earlier quoted context omitted.
That strikes me as a common hugepages win. People never believe you, though, when you say you can make their thing 20% faster for free.
Then it should be pretty easy to display that 20% "faster for free", no? But as always the devil is in the details. I experimented a lot with huge pages, and although in theory you should see the performance boost, the workloads I have been using to test this hypothesis did not end up with anything statistically significant/measurable. So, my conclusion was ... it depends.
Try a big factorio map just as a test case. It's a bit of an outlier on performance, in particular it's very heavy on memory bandwidth.
Re: Meta’s renewed commitment to jemalloc
#238Earlier quoted context omitted.
Then there’s perl, which doesn’t free at all.
Perl frees memory. It uses refcounting, so you need to break heap cycles or it will leak. (99% of the time, I find this less problematic than Java’s approach, fwiw).
Unless this has changed recently, perl doesn't free memory to the kernel, only within its own process/vm.
Re: Meta’s renewed commitment to jemalloc
#239Earlier quoted context omitted.
> A heap uses a heap structure and hops around in memory for every allocation and free. Not in the JVM. And maybe this is ultimately what we are butting up against. After all, the JVM isn't all GCed languages, it's just one of many. In the JVM, heap allocations are done via bump allocation. When a region is filled, the JVM performs a garbage collection which moves objects in the heap (it compacts the memory). It's no…
In the JVM, heap allocations are done via bump allocation. If that were true then they wouldn't be heap allocations. https://www.digitalocean.com/community/tutorials/java-jvm-me... https://docs.oracle.com/en/java/javase/21/core/heap-and-heap... not possible to do in the JVM, barring primitives Then you make data structures out of arrays of primitives. Easy to do? Sure. Easy to do fast? Well, no. That's entirely the r…
> If that were true then they wouldn't be heap allocations.
"Heap" is a misnomer. It's not called that due to the classic CS "heap" datastructure. It's called that for the same reason it's called a heap allocation in C++. Modern C++ allocators don't use a heap structure either.
How the JVM does allocations for all it's collectors is in fact a bump allocator in the heap space. There are some weedsy details (for example, threads in the JVM have their own heap space for doing allocation to avoid contention in allocation) but suffice it to say it ultimately translates into a region check then pointer bump. This is why the JVM is so fast at allocation, much faster than C++ can be. [1] [2]
> I don't know what this means.
JVM allocations are typically pointer bumps, adding a number to a register. There's really nothing faster than it. If you are implementing an arena then you've already lost in terms of performance.
[1] https://www.datadoghq.com/blog/understanding-java-gc/#memory...
Re: Meta’s renewed commitment to jemalloc
#240Earlier quoted context omitted.
[flagged]
Fwiw, this sounds like a healthy discourse - you don’t have to agree on everything, every approach has its merits, code that ends up shipping and supporting production wins the argument in some sense… This is not special to Meta in any way, I observed it in any team which has more than 1 strong senior engineer.
There's nothing healthy about holding on to a work grudge from 10 years ago and then dragging it out in public. That's toxic AF.