Jemalloc and Jalad at Tanagra!
Meta’s renewed commitment to jemalloc
211–220 of 259 posts
Re: Meta’s renewed commitment to jemalloc
#212Earlier quoted context omitted.
Java has a quite strict max heap setting, it's very uncommon to let it allocate up to 25% of the system memory (the default). It won't grow past that point, though. Baring bugs/native leaks - Java has a very predictable memory allocation.
we aren't talking about allocation, tho we are talking about DEallocation
it's a reply to:
"To an outsider, that looks like the JVM heap just steadily growing, which is easy to mistake for a memory leak."
I cut the part that it's possible to make JVM return memory heap after compaction but usually it's not done, i.e. if something grew once, it's likely to do it again.
Re: Meta’s renewed commitment to jemalloc
#213> We plan to deliver improvements to [..] purging mechanisms During my time at Facebook, I maintained a bunch of kernel patches to improve jemalloc purging mechanisms. It wasn't popular in the kernel or the security community, but it was more efficient on benchmarks for sure. Many programs run multiple threads, allocate in one and free in the other. Jemalloc's primary mechanism used to be: madvise the page back to th…
I'm really surprised to see you still hocking this. We did extensive benchmarking of HHVM with and without your patches, and they were proven to make no statistically significant difference in high level metrics. So we dropped them out of the kernel, and they never went back in. I don't doubt for a second you can come up with specific counterexamples and microbenchnarks which show benefit. But you were unable to show…
I wouldn't be surprised if both 'adsharma' and 'jcalvinowens' were right, just at different points in time, perhaps in a bit different context. Things change.
Re: Meta’s renewed commitment to jemalloc
#214Earlier quoted context omitted.
[flagged]
Fwiw, this sounds like a healthy discourse - you don’t have to agree on everything, every approach has its merits, code that ends up shipping and supporting production wins the argument in some sense… This is not special to Meta in any way, I observed it in any team which has more than 1 strong senior engineer.
No, calling out your ex colleague in public years after is not a "healthy discourse" ...
Re: Meta’s renewed commitment to jemalloc
#215Earlier quoted context omitted.
For the peanut gallery more: I worked with both of these guys at Meta on this. The "servers are only on for a few hours" thing was like never true so I have no idea where that claim is coming from. The web performance test took more than a few hours to run alone and we had way more aggressive soaks for other workloads. My recollection was that "write zeroes" just became a cheaper operation between '12 and '14. A fun…
It's not just that zeroing got cheaper, but also we're doing a lot less of it, because jemalloc got much better. If the allocator returns a page to the kernel and then immediately asks back for one, it's not doing its job well: the main purpose of the allocator is to cache allocations from the kernel. Those patches are pre-decay, pre-background purging thread; these changes significantly improve how jemalloc holds on…
I am trying to understand the reason behind why "zeroing got cheaper" circa 2012-2014. Do you have some plausible explanations that you can share?
Haswell (2013) doubled the store throughput to 32 bytes/cycle per core, and Sandy Bridge (2011) doubled the load throughput to the same, but the dataset being operated at FB is most likely much larger than what L1+L2+L3 can fit so I am wondering how much effect the vectorization engine might have had since bulk-zeroing operation for large datasets is anyways going to be bottlenecked by the single core memory bandwidth, which at the time was ~20GB/s.
Perhaps the operation became cheaper simply because of moving to another CPU uarch with higher clock and larger memory bandwidth rather than the vectorization.
Re: Meta’s renewed commitment to jemalloc
#216Earlier quoted context omitted.
So for any process that's using less than 16GB, it's a significant performance boost. And most processes using more RAM, but not splitting accesses across more than 16 zones in rapid succession, will also see a performance boost. My old Intel CPU only has 4 slots for 1GB pages, and that was enough to get me about a 20% performance boost on Factorio. (I think a couple percent might have been allocator change but the b…
That strikes me as a common hugepages win. People never believe you, though, when you say you can make their thing 20% faster for free.
Then it should be pretty easy to display that 20% "faster for free", no? But as always the devil is in the details. I experimented a lot with huge pages, and although in theory you should see the performance boost, the workloads I have been using to test this hypothesis did not end up with anything statistically significant/measurable. So, my conclusion was ... it depends.
Re: Meta’s renewed commitment to jemalloc
#217Earlier quoted context omitted.
> People typically do funky stuff with memory allocation because they have to This same dev did things like putting what he deemed as being large objects (icons) into weak references to save memory. When the references were collected, invariably they had to be reloaded. That was not the source of memory pressure issues in the app. I've developed a mistrust for a lot of devs "doing it because we have to" when it comes…
I'm not sure why you're rationale for how to deal with garbage collected memory is based on a guy that didn't know standard data structures and your own gut feelings. Any program that cares about performance is going to focus on minimizing memory allocation first. The difference between a GCed language like java is that the problems manifest as gc pauses that may or may not be predictable. In a language like C++ you…
Well, let me just circle back to the start of this comment chain.
> Many programs in GC language end up fighting the GC by allocating a large buffer and managing it by hand
That's the primary thing I'm contending with. This is a strategy for fighting the GC, but it's also generally a bad strategy. One that I think gets pulled more because someone heard of the suggestion and less because it's a good way to make things faster.
That guy I'm talking about did a lot of "performance optimizations" based on gut feelings and not data. I've observed that a lot of engineers operate that way.
But I've further observed that when it comes to optimizing for the GC, a large amount of problems don't need such an extreme measure like building your own memory buffer and managing it directly. In fact, that sort of a measure is generally counter productive in a GC environment as it makes major collections more costly. It isn't a "never do this" thing, but it's also not something that "many programs" should be doing.
I agree that many programs with a GC will probably need to change their algorithms to minimize allocations. I disagree that "allocating a large buffer and managing it by hand" is a technique that almost any program or library needs to engage in to minimize GCs.
Re: Meta’s renewed commitment to jemalloc
#218Earlier quoted context omitted.
Unfortunately, the JVM and collectors like the JVM's plays really bad with virtual memory. (Actually, G1 might play better. Everything else does not). The issue is that through the standard course of a JVM application running, every allocated page will ultimately be touched. The JVM fills up new gen, runs a minor collection, moves old objects to old gen, and continues until old gen gets filled. When old gen is filled…
Depends on which JVM, PTC and Aicas do alright with their real time GCs for embedded deployment.
I've never really used anything other than the OpenJDK and Azuls.
How does PTC and Aicas does GC? Is it ref counted? I'm guessing they aren't doing moving collectors.
Re: Meta’s renewed commitment to jemalloc
#219Earlier quoted context omitted.
It wasn't any cgroup. If you put two untrusting processes in a memory cgroup, there is a lot that can go wrong. If you don't like the idea of memory cgroups as a security domain, you could tighten it to be a process. But kernel developers have been opposed to tracking pages on a per address space basis for a long time. On the other hand memory cgroup tracking happens by construction.
> across a process boundary > within a cgroup Note the complementary language usage here. You seem to have interpreted that as me writing that it didn't matter what cgroup they are in, which is an odd thing to claim that I implied. I meant within the same cgroup obviously. Yes, you can read memory out of another process through other means.. but you shouldn't map pages, be able to read them and see what happened in a…
I was clarifying that there are CPU cgroups, network cgroups etc and the proposal touched only memory cgroups.
The people deploying it are free to restrict the cgroup to one process before requesting MAP_UNINITIALIZED if there is a concern around security. At that point the memory cgroup becomes a way to get around the page tracking restriction.
But I get why aesthetically this idea sounds icky to a lot of people.
Re: Meta’s renewed commitment to jemalloc
#220Earlier quoted context omitted.
[flagged]
This is literally how pretty much every conversation goes when you work with people close to the metal. It's a stylistic thing at this point. For what it's worth, 20 years ago all programming newsgroups were like this. I grew my thick skin on alt.lang.perl lol
Except one is an employee and the other one is an ex employee. The bias this introduces is not just a minor nuance, it's what fuels the public conflict and causes everybody else to double check their popcorn reserves.
Of course technical discussions happen all the time at companies between competent people. But you don't do that in public, nor is this a technical debate: "I don't recall talking to you about it" - "I do, I did xyz then you ignored me" - ""