Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta’s renewed commitment to jemalloc

engineering.fb.com

Post reply on HN
Print view View original

141–150 of 259 posts

Re: Meta’s renewed commitment to jemalloc

#141

Post by adsharma »

jcalvinowens wrote: post #138
adsharma wrote: post #72

> We plan to deliver improvements to [..] purging mechanisms During my time at Facebook, I maintained a bunch of kernel patches to improve jemalloc purging mechanisms. It wasn't popular in the kernel or the security community, but it was more efficient on benchmarks for sure. Many programs run multiple threads, allocate in one and free in the other. Jemalloc's primary mechanism used to be: madvise the page back to th…

I'm really surprised to see you still hocking this. We did extensive benchmarking of HHVM with and without your patches, and they were proven to make no statistically significant difference in high level metrics. So we dropped them out of the kernel, and they never went back in. I don't doubt for a second you can come up with specific counterexamples and microbenchnarks which show benefit. But you were unable to show…

You probably weren't there when servers were running for many days at a time.

By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over. If you're restarting the server every few hours, of course the memory fragmentation isn't much of an issue.

> But you were unable to show an advantage at the system level when challenged on it, and that's what matters.

You mean 5 years after I stopped working on the kernel and the underlying system had changed?

I don't recall ever talking to you on the matter.

Re: Meta’s renewed commitment to jemalloc

#142

Post by adsharma »

asveikau wrote: post #140
adsharma wrote: post #72

> We plan to deliver improvements to [..] purging mechanisms During my time at Facebook, I maintained a bunch of kernel patches to improve jemalloc purging mechanisms. It wasn't popular in the kernel or the security community, but it was more efficient on benchmarks for sure. Many programs run multiple threads, allocate in one and free in the other. Jemalloc's primary mechanism used to be: madvise the page back to th…

Maybe I'm misreading, but considering it OK to leak memory contents across a process boundary because it's within a cgroup sounds wild.

It wasn't any cgroup. If you put two untrusting processes in a memory cgroup, there is a lot that can go wrong.

If you don't like the idea of memory cgroups as a security domain, you could tighten it to be a process. But kernel developers have been opposed to tracking pages on a per address space basis for a long time. On the other hand memory cgroup tracking happens by construction.

Re: Meta’s renewed commitment to jemalloc

#143

Post by jcalvinowens »

adsharma wrote: post #141
jcalvinowens wrote: post #138

Earlier quoted context omitted.

I'm really surprised to see you still hocking this. We did extensive benchmarking of HHVM with and without your patches, and they were proven to make no statistically significant difference in high level metrics. So we dropped them out of the kernel, and they never went back in. I don't doubt for a second you can come up with specific counterexamples and microbenchnarks which show benefit. But you were unable to show…

You probably weren't there when servers were running for many days at a time. By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over. If you're restarting the server every few hours, of course the memory fragmentation isn't much of an issue. > But you were unable to show an advantage at the system level when challenged on it, and that's what matters. You mean 5 years af…

> By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over

Nope, I started in 2014.

> I don't recall ever talking to you on the matter.

I recall. You refused to believe the benchmark results and made me repeat the test, then stopped replying after I did :)

Re: Meta’s renewed commitment to jemalloc

#144

Post by adsharma »

jcalvinowens wrote: post #143
adsharma wrote: post #141

Earlier quoted context omitted.

You probably weren't there when servers were running for many days at a time. By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over. If you're restarting the server every few hours, of course the memory fragmentation isn't much of an issue. > But you were unable to show an advantage at the system level when challenged on it, and that's what matters. You mean 5 years af…

> By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over Nope, I started in 2014. > I don't recall ever talking to you on the matter. I recall. You refused to believe the benchmark results and made me repeat the test, then stopped replying after I did :)

The patches were written in 2011 and published in 2012. They did what they were supposed to at the time.

For the peanut gallery: this is a manifestation of an internal eng culture at fb that I wasn't particularly fond of. Celebrating that "I killed X" and partying about it.

You didn't reply to the main point: did you benchmark a server that was running several days at a time? Reasonable people can disagree about whether this a good deployment strategy or not. I tend to believe that there are many places which want to deploy servers and run for months if not days.

Re: Meta’s renewed commitment to jemalloc

#145

Post by nullpoint420 »

jcalvinowens wrote: post #143
adsharma wrote: post #141

Earlier quoted context omitted.

You probably weren't there when servers were running for many days at a time. By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over. If you're restarting the server every few hours, of course the memory fragmentation isn't much of an issue. > But you were unable to show an advantage at the system level when challenged on it, and that's what matters. You mean 5 years af…

> By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over Nope, I started in 2014. > I don't recall ever talking to you on the matter. I recall. You refused to believe the benchmark results and made me repeat the test, then stopped replying after I did :)

This is why I love hacker news. I learn so much from these moments.

Re: Meta’s renewed commitment to jemalloc

#146

Post by alexgartrell »

adsharma wrote: post #144
jcalvinowens wrote: post #143

Earlier quoted context omitted.

> By the time you joined and benchmarked these systems, the continuous rolling deployment had taken over Nope, I started in 2014. > I don't recall ever talking to you on the matter. I recall. You refused to believe the benchmark results and made me repeat the test, then stopped replying after I did :)

The patches were written in 2011 and published in 2012. They did what they were supposed to at the time. For the peanut gallery: this is a manifestation of an internal eng culture at fb that I wasn't particularly fond of. Celebrating that "I killed X" and partying about it. You didn't reply to the main point: did you benchmark a server that was running several days at a time? Reasonable people can disagree about whet…

For the peanut gallery more: I worked with both of these guys at Meta on this.

The "servers are only on for a few hours" thing was like never true so I have no idea where that claim is coming from. The web performance test took more than a few hours to run alone and we had way more aggressive soaks for other workloads.

My recollection was that "write zeroes" just became a cheaper operation between '12 and '14.

A fun fact to distract from the awkwardness: a lot of the kernel work done in the early days was exceedingly scrappy. The port mapping stuff for memcached UDP before SO_REUSEPORT for example. FB binaries couldn't even run on vanilla linux a lot of the time. Over the next several years we put a TON of effort in getting as close to mainline as possible and now Meta is one of the biggest drivers of Linux development.

Re: Meta’s renewed commitment to jemalloc

#147

Post by adsharma »

alexgartrell wrote: post #146
adsharma wrote: post #144

Earlier quoted context omitted.

The patches were written in 2011 and published in 2012. They did what they were supposed to at the time. For the peanut gallery: this is a manifestation of an internal eng culture at fb that I wasn't particularly fond of. Celebrating that "I killed X" and partying about it. You didn't reply to the main point: did you benchmark a server that was running several days at a time? Reasonable people can disagree about whet…

For the peanut gallery more: I worked with both of these guys at Meta on this. The "servers are only on for a few hours" thing was like never true so I have no idea where that claim is coming from. The web performance test took more than a few hours to run alone and we had way more aggressive soaks for other workloads. My recollection was that "write zeroes" just became a cheaper operation between '12 and '14. A fun…

[ Edit: "servers" in this context meant the HHVM server processes, not the physical server which of course had a longer uptime ]

People got promoted for continuous deployment

https://engineering.fb.com/2017/08/31/web/rapid-release-at-m...

I think it's fair to say the hardware changed, the deployment strategy changed and the patches were no longer relevant, so we stopped applying them.

When I showed up, there were 100+ patches on top of a 2009 kernel tree. I reduced the size to about 10 or so critical patches, rebased them at a 6 months cadence over 2-3 years. Upstreamed a few.

Didn't go around saying those old patches were bad ideas and I got rid of them. How you say it matters.

Re: Meta’s renewed commitment to jemalloc

#148

Post by skavi »

usrnm wrote: post #107
skavi wrote: post #61

Earlier quoted context omitted.

modern tcmalloc uses per CPU caches via rseq [0]. We use async rust with multithreaded tokio executors (sometimes multiple in the same application). so relatively high thread counts. [0]: https://github.com/google/tcmalloc/blob/master/docs/design.m...

How do you control which CPU your task resumes on? If you don't then it's still the same problem described above, no?

on the OS scheduler side, i'd imagine there's some stickiness that keeps tasks from jumping wildly between cores. like i'd expect migration to be modelled as a non zero cost. complete speculation though.

tokio scheduler side, the executor is thread per core and work stealing of in progress tasks shouldn't be happening too much.

for all thread pool threads or threads unaffiliated with the executor, see earlier speculation on OS scheduler behavior.

Re: Meta’s renewed commitment to jemalloc

#149

Post by apatheticonion »

Pepe1vo wrote: post #126
apatheticonion wrote: post #117

As an Australian who was just made redundant from a role that involved this type of low level programming - I love working on these these kinds of challenges. I'm saddened that the job market in Australia is largely React CRUD applications and that it's unlikely I will find a role that lets me leverage my niche skill set (which is also my hobby)

Not sure if it's the domain you're interested in, but there are quite a few HFT firms with offices in Australia. The one I know of (IMC trading) does a lot of low level stuff like this and is currently hiring.

I just tried to apply for IMC, the form on their careers page is broken. Looks like that's the first boss to defeat, haha

Re: Meta’s renewed commitment to jemalloc

#150

Post by packetlost »

skavi wrote: post #148
usrnm wrote: post #107

Earlier quoted context omitted.

How do you control which CPU your task resumes on? If you don't then it's still the same problem described above, no?

on the OS scheduler side, i'd imagine there's some stickiness that keeps tasks from jumping wildly between cores. like i'd expect migration to be modelled as a non zero cost. complete speculation though. tokio scheduler side, the executor is thread per core and work stealing of in progress tasks shouldn't be happening too much. for all thread pool threads or threads unaffiliated with the executor, see earlier specula…

Correct. The Linux scheduler has been NUMA aware + sticky for awhile (which is more or less what this reduces to in common scenarios).
Post reply on HN