Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jemalloc Postmortem

jasone.github.io

Post reply on HN
Print view View original

61–70 of 250 posts

Re: Jemalloc Postmortem

#61

Post by MaskRay »

Svetlitski wrote: post #45
kstrauser wrote: post #39

Earlier quoted context omitted.

Stuff like this is what keeps me coming back here. Thanks for posting this! What's hard about using TCMalloc if you're not using bazel? (Not asking to imply that it's not, but because I'm genuinely curious.)

It’s just a huge pain to build and link against. Before the bazel 7.4.0 change your options were basically: 1. Use it as a dynamically linked library. This is not great because you’re taking at a minimum the performance hit of going through the PLT for every call. The forfeited performance is even larger if you compare against statically linking with LTO (i.e. so that you can inline calls to malloc, get the benefit o…

Thanks for sharing the insight!

As I observed when I was at Google: tcmalloc wasn't a dedicated team but a project driven by server performance optimization engineers aiming to improve performance of important internal servers. Extracting it to github.com/google/tcmalloc was complex due to intricate dependencies (https://abseil.io/blog/20200212-tcmalloc ). As internal performance priorities demanded more focus, less time was available for maintaining the CMake build system. Maintaining the repo could at best be described as a community contribution activity.

> Meta’s needs stopped aligning well with those of external uses some time ago, and they are better off doing their own thing.

I think Google's diverged from the external uses even long ago:) (For a long time google3 and gperftools's tcmalloc implementations were so different.)

Re: Jemalloc Postmortem

#63

Post by gilgoomesh »

runevault wrote: post #46
userbinator wrote: post #32

A bad choice of title, as "postmortem" made me think there was some severe outage caused by jemalloc.

postmortem is looking back after an event. That can be a security event/outage, it can also be the completion of a project (see: game studios often do postmortems once their game is out to look back on what went wrong and right between preproduction, production, and post launch).

It's weird that we use "postmortem" in those cases since the word literally means "after death"; kind of implying something bad happened. I get that most of these postmortems are done after major development ceases, so it kind of is "dead" but still.

Surely a "retrospective" would be a better word for a look back. It even means "look back.

Re: Jemalloc Postmortem

#64

Post by glandium »

meisel wrote: post #19

I believe there’s no other allocator besides jemalloc that can seamlessly override macOS malloc/free like people do with LD_PRELOAD on Linux (at least as of ~2020). jemalloc has a very nice zone-based way of making itself the default, and manages to accommodate Apple’s odd requirements for an allocator that have tripped other third-party allocators up when trying to override malloc/free.

Note this requires hackery that relies on Apple not changing things in its system allocator, which has happened at least twice IIRC.

Re: Jemalloc Postmortem

#65

Post by masklinn »

> jemalloc was probably booted from Rust binaries sooner than the natural course of development might have otherwise dictated.

FWIW while it was a factor it was just one of a number: https://github.com/rust-lang/rust/issues/36963#issuecomment-...

And jemalloc was only removed two years after that issue was opened: https://github.com/rust-lang/rust/pull/55238

Re: Jemalloc Postmortem

#66

Post by jdsully »

vlovich123 wrote: post #60
favorited wrote: post #41

Earlier quoted context omitted.

It's possible that they were referring to something specific about their platform and its system allocator, but like I said it was an anecdote about one engineer's statement. I just remember thinking it sounded fair at the time.

The “system” allocator is managing memory within a process boundary. The kernel is responsible for managing it across processes. Claiming that a user space allocator is greedily inefficient is voodoo reasoning that suggests the person making the claim has a poor grasp of architecture.

The "greedy" part is likely not releasing pages back to the OS in a timely manner.

Re: Jemalloc Postmortem

#67

Post by Twirrim »

dpe82 wrote: post #5
Twirrim wrote: post #3

Oh that's interesting. jemalloc is the memory allocator used by redis, among other projects. Wonder what the performance impact will be if they have to change allocators.

Why would they have to change? Sometimes software development is largely "done" and there isn't much more you need to do to a library.

While I certainly wish that more software would reach a "done" stage, I don't think jemalloc is necessarily there yet. Unfortunately I'm aware of there being bugs in the current version of jemalloc, when run in certain environment configurations, including memory leaks. I know the folks that found it were looking to report it, but I guess that won't happen now.

Even from a quick look at the open issues, I can see https://github.com/jemalloc/jemalloc/issues/2838, and https://github.com/jemalloc/jemalloc/issues/2815 as two examples, but there's a fair number of issues still open against the repository.

So that'll leave projects like redis & valkey with some decisions to make.

1) Keep jemalloc and accept things like memory leak bugs

2) Fork and maintain their own version of jemalloc.

3) Spend time replacing it entirely.

4) Hope someone else picks it up?

Re: Jemalloc Postmortem

#68

Post by jeffbee »

vlovich123 wrote: post #60
favorited wrote: post #41

Earlier quoted context omitted.

It's possible that they were referring to something specific about their platform and its system allocator, but like I said it was an anecdote about one engineer's statement. I just remember thinking it sounded fair at the time.

The “system” allocator is managing memory within a process boundary. The kernel is responsible for managing it across processes. Claiming that a user space allocator is greedily inefficient is voodoo reasoning that suggests the person making the claim has a poor grasp of architecture.

There are shared resources involved though, for example one process can cause a lot of traffic in khugepaged. However I would point out that is an endemic risk of Linux's overall architecture. Any process can cause chaos by dirtying pages, or otherwise triggering reclaim.

Re: Jemalloc Postmortem

#69

Post by Twirrim »

jeffbee wrote: post #7
dpe82 wrote: post #5

Earlier quoted context omitted.

Why would they have to change? Sometimes software development is largely "done" and there isn't much more you need to do to a library.

For an example of why an allocator is a maintenance treadmill, consider that C++ recently (relatively) added sized delete, and Linux recently gained transparent huge pages.

It's been 14 years since THP got added to the kernel[1], surely we're past calling that "recent" :)

https://www.kernelconfig.io/config_transparent_hugepage

Re: Jemalloc Postmortem

#70

Post by masklinn »

vlovich123 wrote: post #57
mavis wrote: post #37

Switching to jemalloc instantly fixed an irksome memory leak in an embedded Linux appliance I inherited many moons ago. Thank you je, we salute you!

That’s because sane allocators that aren’t glibc will return unused memory periodically to the OS while glibc prefers to permanently retain said memory.

glibc will return memory to the OS just fine, the problem is that its arena design is extremely prone to fragmentation, so you end up with a bunch of arenas which are almost but not quite empty and can't be released, but can’t really be used either.

In fact, Jason himself (the author of jemalloc and TFA) posted an article on glibc malloc fragmentation 15 years ago: https://web.archive.org/web/20160417080412/http://www.canonw...

And it's an issue to this day: https://blog.arkey.fr/drafts/2021/01/22/native-memory-fragme...

Post reply on HN