Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jemalloc Postmortem

jasone.github.io

Post reply on HN
Print view View original

41–50 of 250 posts

Re: Jemalloc Postmortem

#41

Post by favorited »

jeffbee wrote: post #31
favorited wrote: post #30

Earlier quoted context omitted.

Disclaimer: I'm not an allocator engineer, this is just an anecdote. A while back, I had a conversation with an engineer who maintained an OS allocator, and their claim was that custom allocators tend to make one process's memory allocation faster at the expense of the rest of the system. System allocators are less able to make allocation fair holistically, because one process isn't following the same patterns as the…

I don't think that's really a position that can be defended. Both jemalloc and tcmalloc evolved and were refined in antagonistic multitenant environments without one overwhelming application. They are optimal for that exact thing.

It's possible that they were referring to something specific about their platform and its system allocator, but like I said it was an anecdote about one engineer's statement. I just remember thinking it sounded fair at the time.

Re: Jemalloc Postmortem

#42

Post by kstrauser »

sanxiyn wrote: post #24
kstrauser wrote: post #22

I’ve wondered about this before but never when around people who might know. From my outsider view, jemalloc looked like a strict improvement over glibc’s malloc, according to all the benchmarks I’d seen when the subject came up. So, why isn’t it the default allocator?

As far as I know there is no technical reason why jemalloc shouldn't be the default allocator. In fact, as pointed out in the article, it IS the default allocator on FreeBSD. My understanding is it is largely political.

Now that I think about it, I could easily imagine it being left out of glibc because it doesn't build on Hurd or something.

Re: Jemalloc Postmortem

#43

Post by kstrauser »

gdiamos wrote: post #2

Congrats on the great run and the future. Jemalloc was an inspirational to many memory allocators.

I was using FreeBSD back when jemalloc came along, and it blew my mind to imagine swapping out just that one (major) part of its libc. Honestly, it hadn't occured to me, and made me wonder what else we could wholesale replace.

Re: Jemalloc Postmortem

#44

Post by kstrauser »

skeptrune wrote: post #23

Kind of nuts that he worked on Jemalloc for over a decade while having personal preference for garbage collection. I'm surprised he doesn't have more regret.

Why are those two mutually exclusive? I'd think that a high performance allocator would be especially crucial in the implementation of a fast garbage collected language. For example, in Python you can't alloc(n * sizeof(obj)) to reserve that much contiguous space for n objects. Instead, you use the builtins which isolate you from that low-level bookkeeping. Those builtins have to be pretty fast or performance would be terrible.

Re: Jemalloc Postmortem

#45

Post by Svetlitski »

kstrauser wrote: post #39
Svetlitski wrote: post #36

I understand the decision to archive the upstream repo; as of when I left Meta, we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file (my favorite was the time someone filed an issue because our test suite didn’t pass on Itanium lol). Still, it makes me sad to see. Jemalloc is still IMO the best-performing general-purpose malloc implementation that’s…

Stuff like this is what keeps me coming back here. Thanks for posting this! What's hard about using TCMalloc if you're not using bazel? (Not asking to imply that it's not, but because I'm genuinely curious.)

It’s just a huge pain to build and link against. Before the bazel 7.4.0 change your options were basically:

1. Use it as a dynamically linked library. This is not great because you’re taking at a minimum the performance hit of going through the PLT for every call. The forfeited performance is even larger if you compare against statically linking with LTO (i.e. so that you can inline calls to malloc, get the benefit of FDO , etc.). Not to mention all the deployment headaches associated with shared libraries.

2. Painfully manually create a static library. I’ve done this, it’s awful; especially if you want to go the extra mile to capture as much performance as possible and at least get partial LTO (i.e. of TCMalloc independent of your application code, compiling all of TCMalloc’s compilation units together to create a single object file).

When I was at Meta I imported TCMalloc to benchmark against (to highlight areas where we could do better in Jemalloc) by pain-stakingly hand-translating its bazel BUILD files to buck2 because there was legitimately no better option.

As a consequence of being so hard to use outside of Google, TCMalloc has many more unexpected (sometimes problematic) behaviors than Jemalloc when used as a general purpose allocator in other environments (e.g. it basically assumes that you are using a certain set of Linux configuration options [1] and behaves rather poorly if you’re not)

[1] https://google.github.io/tcmalloc/tuning.html#system-level-o...

Re: Jemalloc Postmortem

#46

Post by runevault »

userbinator wrote: post #32

A bad choice of title, as "postmortem" made me think there was some severe outage caused by jemalloc.

postmortem is looking back after an event. That can be a security event/outage, it can also be the completion of a project (see: game studios often do postmortems once their game is out to look back on what went wrong and right between preproduction, production, and post launch).

Re: Jemalloc Postmortem

#47

Post by kstrauser »

Svetlitski wrote: post #45
kstrauser wrote: post #39

Earlier quoted context omitted.

Stuff like this is what keeps me coming back here. Thanks for posting this! What's hard about using TCMalloc if you're not using bazel? (Not asking to imply that it's not, but because I'm genuinely curious.)

It’s just a huge pain to build and link against. Before the bazel 7.4.0 change your options were basically: 1. Use it as a dynamically linked library. This is not great because you’re taking at a minimum the performance hit of going through the PLT for every call. The forfeited performance is even larger if you compare against statically linking with LTO (i.e. so that you can inline calls to malloc, get the benefit o…

Wow. That does sound quite unpleasant.

Thanks again. This is far outside my regular work, but it fascinates me.

Re: Jemalloc Postmortem

#48

Post by matoro »

Svetlitski wrote: post #36

I understand the decision to archive the upstream repo; as of when I left Meta, we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file (my favorite was the time someone filed an issue because our test suite didn’t pass on Itanium lol). Still, it makes me sad to see. Jemalloc is still IMO the best-performing general-purpose malloc implementation that’s…

That was me that filed the Itanium test suite failure. :)

Re: Jemalloc Postmortem

#49

Post by EnPissant »

Svetlitski wrote: post #36

I understand the decision to archive the upstream repo; as of when I left Meta, we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file (my favorite was the time someone filed an issue because our test suite didn’t pass on Itanium lol). Still, it makes me sad to see. Jemalloc is still IMO the best-performing general-purpose malloc implementation that’s…

Do you have any opinions on mimalloc?

Re: Jemalloc Postmortem

#50

Post by Svetlitski »

chubot wrote: post #20

Nice post -- so does Facebook no longer use jemalloc at all? Or is it maintenance mode? Or I wonder if they could simply use tcmalloc or another allocator these days? Facebook infrastructure engineering reduced investment in core technology, instead emphasizing return on investment.

As of when I left Meta nearly two years ago (although I would be absolutely shocked if this isn’t still the case) Jemalloc is the allocator, and is statically linked into every single binary running at the company.

> Or I wonder if they could simply use tcmalloc or another allocator these days?

Jemalloc is very deeply integrated there, so this is a lot harder than it sounds. From the telemetry being plumbed through in Strobelight, to applications using every highly Jemalloc-specific extension under the sun (e.g. manually created arenas with custom extent hooks), to the convergent evolution of applications being written in ways such that they perform optimally with respect to Jemalloc’s exact behavior.

Post reply on HN