181–190 of 259 posts
Earlier quoted context omitted.
This is why I love hacker news. I learn so much from these moments.
Like "never work at Meta unless you can out-toxic your coworkers".
As an Australian who was just made redundant from a role that involved this type of low level programming - I love working on these these kinds of challenges. I'm saddened that the job market in Australia is largely React CRUD applications and that it's unlikely I will find a role that lets me leverage my niche skill set (which is also my hobby)
I know this isn’t who’s hiring thread, but we are hiring in AU for low-level data processing and have interesting performance challenges. Link in bio.
[dead]
Earlier quoted context omitted.
The patches were written in 2011 and published in 2012. They did what they were supposed to at the time. For the peanut gallery: this is a manifestation of an internal eng culture at fb that I wasn't particularly fond of. Celebrating that "I killed X" and partying about it. You didn't reply to the main point: did you benchmark a server that was running several days at a time? Reasonable people can disagree about whet…
For the peanut gallery more: I worked with both of these guys at Meta on this. The "servers are only on for a few hours" thing was like never true so I have no idea where that claim is coming from. The web performance test took more than a few hours to run alone and we had way more aggressive soaks for other workloads. My recollection was that "write zeroes" just became a cheaper operation between '12 and '14. A fun…
If the allocator returns a page to the kernel and then immediately asks back for one, it's not doing its job well: the main purpose of the allocator is to cache allocations from the kernel. Those patches are pre-decay, pre-background purging thread; these changes significantly improve how jemalloc holds on to memory that might be needed soon. Instead, the zeroing out patches optimize for the pathological behavior.
Also, the kernel has since exposed better ways to optimize memory reclamation, like MADV_FREE, which is a "lazy reclaim": the page stays mapped to the process until the kernel actually need it, so if we use it again before that happens, the whole unmapping/mapping is avoided, which saves not only the zeroing cost, but also the TLB shootdown and other costs. And without changing any security boundary. jemalloc can take advantage of this by enabling "muzzy decay".
However, the drawback is that system-level memory accounting becomes even more fuzzy.
(hi Alex!)
If you need to optimize the allocator you are doing it wrong.
A 0.5% improvement may not be a lot to you, but at hyperscaler scale it's well worth staffing a team to work on it, with the added benefit of having people on hand that can investigate subtle bugs and pathological perf behaviors.
Earlier quoted context omitted.
[ Edit: "servers" in this context meant the HHVM server processes, not the physical server which of course had a longer uptime ] People got promoted for continuous deployment https://engineering.fb.com/2017/08/31/web/rapid-release-at-m... I think it's fair to say the hardware changed, the deployment strategy changed and the patches were no longer relevant, so we stopped applying them. When I showed up, there were 100…
The linked article says they decided to do CD in 2016 fwiw so that's not inconsistent with what I said. You reduced the number of patches a lot and also pushed very hard to get us to 3.0 after we sat on 2.6.38 ~forever. Which was very appreciated, btw. We built the whole plan going forward based on this work. I'm not arguing that anyone should be nice to anyone or not (it's a waste of breath when it comes to Linux).…
Earlier quoted context omitted.
For the peanut gallery more: I worked with both of these guys at Meta on this. The "servers are only on for a few hours" thing was like never true so I have no idea where that claim is coming from. The web performance test took more than a few hours to run alone and we had way more aggressive soaks for other workloads. My recollection was that "write zeroes" just became a cheaper operation between '12 and '14. A fun…
[flagged]
This is not special to Meta in any way, I observed it in any team which has more than 1 strong senior engineer.
Earlier quoted context omitted.
For the peanut gallery more: I worked with both of these guys at Meta on this. The "servers are only on for a few hours" thing was like never true so I have no idea where that claim is coming from. The web performance test took more than a few hours to run alone and we had way more aggressive soaks for other workloads. My recollection was that "write zeroes" just became a cheaper operation between '12 and '14. A fun…
That is, wow, a story. At what point did you realize how different fb engineering was from what you expected?
An important nuance - most Facebook engineers don't believe that Facebook/Meta would continue to grow next year; and that disbelief had been there since as early as in 2018 (when I'd joined).
very few facebook employees use their products outside of testing, which is a big contributor to that fear - they just can't believe that there are billions of people who would continue to use apps to post what they had for lunch!
And as a result of that lack of faith, most of them believe that Meta is a bubble and can burst at any point. Consequently, everyone works for the next performance review cycle, and most are just in rush to capture as much money as they could before that bubble bursts.
Earlier quoted context omitted.
This is why I love hacker news. I learn so much from these moments.
Like "never work at Meta unless you can out-toxic your coworkers".
This thread would've been way more fun with a couple of middle managers and product managers in the mix ;-)