FTA:
> And people find themselves in impossible situations where the main choices are 1) make poor decisions under extreme pressure, 2) comply under extreme pressure, or 3) get routed around.
It doesn't sound like a work place :-(
Jemalloc Postmortem
91–100 of 250 posts
Re: Jemalloc Postmortem
#92Jason, here is a story about how much your work impacts us. We run a decently sized company that processes hundreds of millions of images/videos per day. When we first started about 5 years ago, we spent countless hours debugging issues related to memory fragmentation. One fine day, we discovered Jemalloc and put it in our service, which was causing a lot of memory fragmentation. We did not think that those 2 lines o…
I really don't mean to be snarky, but honest question:
Did you donate? Nothing says thank you like some $$$...
Re: Jemalloc Postmortem
#93Earlier quoted context omitted.
I don't think that's really a position that can be defended. Both jemalloc and tcmalloc evolved and were refined in antagonistic multitenant environments without one overwhelming application. They are optimal for that exact thing.
> Both jemalloc and tcmalloc evolved and were refined in antagonistic multitenant environments without one overwhelming application. They are optimal for that exact thing. They were mostly optimised on Facebook/Google server-side systems, which were likely one application per VM, no? (Unlike desktop usage where users want several applications to run cooperatively). Firefox is a different case but apparently mainline…
Google runs dozens to hundreds of unrelated workloads in lightweight containers on a single machine, in "borg". Facebook has a thing called "tupperware" with the same property.
Re: Jemalloc Postmortem
#94Jason, here is a story about how much your work impacts us. We run a decently sized company that processes hundreds of millions of images/videos per day. When we first started about 5 years ago, we spent countless hours debugging issues related to memory fragmentation. One fine day, we discovered Jemalloc and put it in our service, which was causing a lot of memory fragmentation. We did not think that those 2 lines o…
indeed! most image processing golang services suggest/use jemalloc
the top 3 from https://github.com/topics/resize-images (as of 2025-06-13)
imaginary: https://github.com/h2non/imaginary/blob/1d4e251cfcd58ea66f83...
imgproxy: https://web.archive.org/web/20210412004544/https://docs.imgp... (linked from a discussion in the imaginary repo)
imagor: https://github.com/cshum/imagor/blob/f6673fa6656ee8ef17728f2...
Re: Jemalloc Postmortem
#95Earlier quoted context omitted.
postmortem is looking back after an event. That can be a security event/outage, it can also be the completion of a project (see: game studios often do postmortems once their game is out to look back on what went wrong and right between preproduction, production, and post launch).
It's weird that we use "postmortem" in those cases since the word literally means "after death"; kind of implying something bad happened. I get that most of these postmortems are done after major development ceases, so it kind of is "dead" but still. Surely a "retrospective" would be a better word for a look back. It even means "look back.
It gets even better. Some companies use "mid-mortems", which are evaluation and reflection processes in the middle of a project...
Re: Jemalloc Postmortem
#96> jemalloc was probably booted from Rust binaries sooner than the natural course of development might have otherwise dictated. FWIW while it was a factor it was just one of a number: https://github.com/rust-lang/rust/issues/36963#issuecomment-... And jemalloc was only removed two years after that issue was opened: https://github.com/rust-lang/rust/pull/55238
Interesting that one of the factor listed in there, the hardcoded page-size on arm64, is still is an unsolved issue upstream, and that forces app developers to either ship multiple arm64 linux binaries, or drop support for some platforms.
I wonder if some kind of dynamic page-size (with dynamic ftrace-style binary patching for performance?) would have been that much slower.
Re: Jemalloc Postmortem
#97Re: Jemalloc Postmortem
#98Earlier quoted context omitted.
Stuff like this is what keeps me coming back here. Thanks for posting this! What's hard about using TCMalloc if you're not using bazel? (Not asking to imply that it's not, but because I'm genuinely curious.)
It’s just a huge pain to build and link against. Before the bazel 7.4.0 change your options were basically: 1. Use it as a dynamically linked library. This is not great because you’re taking at a minimum the performance hit of going through the PLT for every call. The forfeited performance is even larger if you compare against statically linking with LTO (i.e. so that you can inline calls to malloc, get the benefit o…
Everything from Google is an absolute pain to work with unless you're in Google using their systems, FWIW. Anything from the Chromium project is deeply intangled with everything else from the Chromium project as part of one gigantic Chromium source tree with all dependencies and toolchains vendored. They do not care about ABI what so ever, to the point that a lot of Google libraries change their public ABI based on whether address sanitizer is enabled or not, meaning you can't enable ASAN for your code if you use pre-built (e.g package manager provided) versions of their code. Their libraries also tend to break if you link against them from a project with RTTI enabled, a compiler set to a slightly different compiler version, or any number of other minute differences that most other developers don't let affect their ABI.
And if you try to build their libraries from source, that involves downloading tens of gigabytes of sysroots and toolchains and vendored dependencies.
Oh and you probably don't want multiple versions of a library in your binary, so be prepared to use Google's (probably outdated) version of whatever libraries they vendor.
And they make no effort what so ever to distinguish between public header files and their source code, so if you wanna package up their libraries, be prepared to make scripts to extract the headers you need (including headers from vendored dependencies), you can't just copy all of some 'include/' folder.
And their public headers tend to do idiotic stuff like `#include "base/pc.h"`, where that `"base/pc.h"` path is not relative to the file doing the include. So you're gonna have to pollute the include namespace. Make sure not to step on their toes! There's a lot of them.
I have had the misfortune of working with Abseill, their WebRTC library, their gRPC library and their protobuf library, and it's all terrible. For personal projects where I don't have a very, very good reason to use Google code, I try to avoid it like the plague. For professional projects where I've had to use libwebrtc, the only reasonable approach is to silo off libwebrtc into its own binary which only deals with WebRTC, typically with a line-delimited JSON protocol on stdin/stdout. For things like protobuf/gRPC where that hasn't been possible, you just have to live with the suffering.
..This comment should probably have been a blog post.
Re: Jemalloc Postmortem
#99I understand the decision to archive the upstream repo; as of when I left Meta, we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file (my favorite was the time someone filed an issue because our test suite didn’t pass on Itanium lol). Still, it makes me sad to see. Jemalloc is still IMO the best-performing general-purpose malloc implementation that’s…
> TCMalloc is great, but is an absolute nightmare to use if you’re not using bazel
custom-malloc-newbie question: Why is the choice of build system (generator) significant when evaluating the usability of a library?
Re: Jemalloc Postmortem
#100Jason, here is a story about how much your work impacts us. We run a decently sized company that processes hundreds of millions of images/videos per day. When we first started about 5 years ago, we spent countless hours debugging issues related to memory fragmentation. One fine day, we discovered Jemalloc and put it in our service, which was causing a lot of memory fragmentation. We did not think that those 2 lines o…
I really don't mean to be snarky, but honest question: Did you donate? Nothing says thank you like some $$$...
It was a meta project and development ceased. For a regular project that expectation is fine, but here it does not apply IMHO.