Live data from Hacker News

Jemalloc Postmortem

jasone.github.io

191–200 of 250 posts

Re: Jemalloc Postmortem

#191
post #3

Oh that's interesting. jemalloc is the memory allocator used by redis, among other projects. Wonder what the performance impact will be if they have to change allocators.

Last I checked Redis used their own fork of jemalloc. It may not even be updated to the latest release.

Re: Jemalloc Postmortem

#192

Earlier quoted context omitted.

Meta has a fork that they still are working on, where development is continuing. https://github.com/facebook/jemalloc

They take everything FLOSS and ruin it with bureaucracy, churn, breakage, and inconsideration to external use. They may claim FOSS broadly but it's mostly FOSS-washed, unusable garbage except for a few popular things.

React, PyTorch, and RocksDB are all extremely significant. Not to mention them being one of the biggest contributors to the Linux kernel.

Re: Jemalloc Postmortem

#193
post #98

Earlier quoted context omitted.

Everything from Google is an absolute pain to work with unless you're in Google using their systems, FWIW. Anything from the Chromium project is deeply intangled with everything else from the Chromium project as part of one gigantic Chromium source tree with all dependencies and toolchains vendored. They do not care about ABI what so ever, to the point that a lot of Google libraries change their public ABI based on w…

I think your rant isn't long enough to include everything relevant ;) The Blink web engine (which I sometimes compile for qtwebengine) takes a really long time to compile, several times longer than Gecko according to some info I found online. Google has a policy of not using forward declarations, including everything instead. That's a pretty big WTF for anyone who has ever optimized build time. Google probably just t…

Chromium deviates from Google-wide policy and allows forward-declarations: https://chromium.googlesource.com/chromium/src/+/main/styleg..., "Forward declarations vs. #includes".

Re: Jemalloc Postmortem

#194

Earlier quoted context omitted.

I think your rant isn't long enough to include everything relevant ;) The Blink web engine (which I sometimes compile for qtwebengine) takes a really long time to compile, several times longer than Gecko according to some info I found online. Google has a policy of not using forward declarations, including everything instead. That's a pretty big WTF for anyone who has ever optimized build time. Google probably just t…

Chromium deviates from Google-wide policy and allows forward-declarations: https://chromium.googlesource.com/chromium/src/+/main/styleg... , "Forward declarations vs. #includes".

That is really nice to hear, but AFAICS it only means that it may change in the future. Because in current code, it was ~all includes last time I checked.

Well, I remember one - very biased - example where I had a look at a class that was especially expensive to compile, like 40 seconds (on a Ryzen 7950X) and maybe 2 GB of RAM. It had under 200 LOC and didn't seem to do anything that's typically expensive to compile... except for the stuff it included. Which also didn't seem to do anything fancy. But transitive includes can snowball if you don't add any "compile firewalls".

Re: Jemalloc Postmortem

#195
post #185
post #166

Earlier quoted context omitted.

Big evil FSF, always trying to extract value and increase their stock price.

That was tongue-in-cheek, I thought Darth Vader's voice was enough of a cue. License switches do happen though, and are the source of outrage. Cue redis. The cause of transferring copyright is often practical (hard to track down + reach out to + gather answers from all authors-slash-contributors which hampers some critical decisions down the road); for the FSF it's ideological (GCC source code must remain under sole…

Apologies. I am generally against CLAs, and I think it's shitty of GNU/FSF to use them, even if they promise to only do good and free things.

Re: Jemalloc Postmortem

#196

Looking at all the comments and lightly browsing the source code, I'm amazed. Both at how much impact a memory allocator can make, but also how much code is involved. I'm not really sure what I expected, but somehow I expect a memory allocator to be ... smaller, simpler perhaps?

mimalloc is cleaner but lacks the very useful profiling features. To be fair it also has not gone through decades of changes as described in the postmortem either.

Re: Jemalloc Postmortem

#197

I understand the decision to archive the upstream repo; as of when I left Meta, we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file (my favorite was the time someone filed an issue because our test suite didn’t pass on Itanium lol). Still, it makes me sad to see. Jemalloc is still IMO the best-performing general-purpose malloc implementation that’s…

It's kind of wild that great software is hindered by a complicated build and integration process.

Re: Jemalloc Postmortem

#198

Earlier quoted context omitted.

Chromium deviates from Google-wide policy and allows forward-declarations: https://chromium.googlesource.com/chromium/src/+/main/styleg... , "Forward declarations vs. #includes".

That is really nice to hear, but AFAICS it only means that it may change in the future. Because in current code, it was ~all includes last time I checked. Well, I remember one - very biased - example where I had a look at a class that was especially expensive to compile, like 40 seconds (on a Ryzen 7950X) and maybe 2 GB of RAM. It had under 200 LOC and didn't seem to do anything that's typically expensive to compile.…

This is actually tracked at a publicly visible URL: https://commondatastorage.googleapis.com/chromium-browser-cl...

And the include graph analysis: https://commondatastorage.googleapis.com/chromium-browser-cl...

The annotated red dots correspond to the last time Chrome developers did a big push to prune the include graph to optimize build time. It was effective, but there was push back. C++ developers just want magic, they don't want to think about dependency management, and it's hard to blame them. But, at the end of the day, builds scale with sources times dependencies, and if you aren't disciplined, you can expect superlinear build times.

Re: Jemalloc Postmortem

#199
post #110
post #107

Earlier quoted context omitted.

Dunno why you got downvoted, but I've also tried to let Claude translate a bunch of BUILD files to equivalent CMakeLists.txt. It worked. The resulting CMakeLists.txt looks super terrible, but so is 95% of CMakeLists.txt in this world, so why bother, it's doomed anyway.

They got downvoted because 1) comments of the form "I gave a chat bot a toy example of a task and it managed it" are tired and uninformative, and 2) because nobody was talking about anything which would make translating a Makefile into Bazel somehow relevant, nobody here has a Makefile which we wish was Bazel, we wish Google code was easier to work with

People are discussing things that are tedious work. I think the conversion to Bazel from a makefile is much more tedious and error prone than the reverse, in part because of Bazel sandboxing although that shouldn’t make much of a difference for a well-defined collection of Makefiles of a C library.

The reverse should be much easier, which was the point of the post. Pointing it out as a capability (translation of build systems) that is handled well, is, well, informative. The future isn’t evenly distributed and people aren’t always aware of capabilities, even on HN

Re: Jemalloc Postmortem

#200

I've used jemalloc in every game engine I've written for years. It's just the thing to do. WAY faster on win32 than the default allocator. It's also nice to have the same allocator across all platforms. I learned of it from it's integration in FreeBSD and never looked back. jemalloc has help entertained a lot of people :)

+1 windows def allocator is pos. Jemalloc rules

>windows def allocator is pos

Wow, still? I remember allocator benchmarks from 10-15 years ago where there were some notable differences between allocators... and then Windows with like 20% the performance of everything else!

Post reply on HN