Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jemalloc Postmortem

jasone.github.io

Post reply on HN
Print view View original

151–160 of 250 posts

Re: Jemalloc Postmortem

#151

Post by nh2 »

charcircuit wrote: post #58
chubot wrote: post #20

Nice post -- so does Facebook no longer use jemalloc at all? Or is it maintenance mode? Or I wonder if they could simply use tcmalloc or another allocator these days? Facebook infrastructure engineering reduced investment in core technology, instead emphasizing return on investment.

Meta has a fork that they still are working on, where development is continuing. https://github.com/facebook/jemalloc

The point of the blog post is that repo is over-focused on Facebook's needs instead of "general utility":

> as a result of recent changes within Meta we no longer have anyone shepherding long-term jemalloc development with an eye toward general utility

> we reached a sad end for jemalloc in the hands of Facebook/Meta

> Meta’s needs stopped aligning well with those of external uses some time ago, and they are better off doing their own thing.

Re: Jemalloc Postmortem

#152

Post by igrunert »

burnt-resistor wrote: post #144

Lesson: Don't let one megacorp dominate or take over your FOSS project. Push back somewhat and say "no" to too much help from one source.

I think the author was happy to be employed by a megacorp, along with a team to push jemalloc forward.

He and the other previous contributors are free to find new employers to continue such an arrangement, if any are willing to make that investment. Alternatively they could cobble together funding from a variety of smaller vendors. I think the author is happy to move on to other projects, after spending a long time in this problem space.

I don’t think that “don’t let one megacorp hire a team of contributors for your FOSS project” is the lesson here. I’d say it’s a lesson in working upstream - the contributions made during their Facebook / Meta investment are available for the community to build upon. They could’ve just as easily been made in a closed source fork inside Facebook, without violating the terms of the license.

Also Mozilla were unable to switch from their fork to the upstream version, and didn’t easily benefit from the Facebook / Meta investment as a result.

Re: Jemalloc Postmortem

#153

Post by nh2 »

nh2 wrote: post #151
charcircuit wrote: post #58

Earlier quoted context omitted.

Meta has a fork that they still are working on, where development is continuing. https://github.com/facebook/jemalloc

The point of the blog post is that repo is over-focused on Facebook's needs instead of "general utility": > as a result of recent changes within Meta we no longer have anyone shepherding long-term jemalloc development with an eye toward general utility > we reached a sad end for jemalloc in the hands of Facebook/Meta > Meta’s needs stopped aligning well with those of external uses some time ago, and they are better o…

But I'd like to know exactly what that means.

How can I find out if Facebook's focus is aligned with my own needs?

Re: Jemalloc Postmortem

#155

Post by jeffbee »

nicoburns wrote: post #120
jdsully wrote: post #66

Earlier quoted context omitted.

The "greedy" part is likely not releasing pages back to the OS in a timely manner.

That seems odd though, seeing as this is one of the main criticisms of glibc's allocator.

In the containerized environments where these allocators were mainly developed, it is all but totally pointless to return memory to the kernel. You might as well keep everything your container is entitled to use, because it's not like the other containers can use it. Someone or some automatic system has written down how much memory the container is going to use.

Re: Jemalloc Postmortem

#156

Post by jeffbee »

mort96 wrote: post #110
rfoo wrote: post #107

Earlier quoted context omitted.

Dunno why you got downvoted, but I've also tried to let Claude translate a bunch of BUILD files to equivalent CMakeLists.txt. It worked. The resulting CMakeLists.txt looks super terrible, but so is 95% of CMakeLists.txt in this world, so why bother, it's doomed anyway.

They got downvoted because 1) comments of the form "I gave a chat bot a toy example of a task and it managed it" are tired and uninformative, and 2) because nobody was talking about anything which would make translating a Makefile into Bazel somehow relevant, nobody here has a Makefile which we wish was Bazel, we wish Google code was easier to work with

The person above was saying they did a tedious manual port of tcmalloc to buck. Since tcmalloc provides both bazel and cmake builds, it seems relevant that in these days a person could have potentially forced a robot to do the job of writing the buck file given the cmake or bazel files.

Re: Jemalloc Postmortem

#157

Post by bluGill »

fc417fc802 wrote: post #113
mort96 wrote: post #98

Earlier quoted context omitted.

Everything from Google is an absolute pain to work with unless you're in Google using their systems, FWIW. Anything from the Chromium project is deeply intangled with everything else from the Chromium project as part of one gigantic Chromium source tree with all dependencies and toolchains vendored. They do not care about ABI what so ever, to the point that a lot of Google libraries change their public ABI based on w…

Reading this perspective was interesting. I can appreciate that things didn't fit into your workflow very well, but my experience has been the opposite. Their projects seem to be structured from the perspective of building literally everything from source on the spot. That matches my mindset - I choose to build from scratch in a network isolated environment. As a result google repos are some of the few that I can cou…

> I choose to build from scratch in a network isolated environment. As a result google repos are some of the few that I can count on to be fairly easy to get up and running.

If you are building a single google project they are easy to get up and running. If you are building your own project on top of theirs things get difficult. those library issues will get you.

I don't know about OP, but we have our own in house package manager. If Conan was ready a couple years sooner we would have used that instead.

Re: Jemalloc Postmortem

#158

Post by jeffbee »

p0w3n3d wrote: post #53

Thank you. Jemalloc was recently recommended to me on some presentation about Java optimization. I wonder if you did get everything you should from the companies that use it. I mean sometimes I feel that big tech firms only use free software, never giving anything to it, so I hope you were the exception here.

Imagine being a Java developer and thinking "what have big tech corporations ever done for me?"

Re: Jemalloc Postmortem

#160

Post by meepmorp »

simonask wrote: post #95
gilgoomesh wrote: post #63

Earlier quoted context omitted.

It's weird that we use "postmortem" in those cases since the word literally means "after death"; kind of implying something bad happened. I get that most of these postmortems are done after major development ceases, so it kind of is "dead" but still. Surely a "retrospective" would be a better word for a look back. It even means "look back.

It gets even better. Some companies use "mid-mortems", which are evaluation and reflection processes in the middle of a project...

sounds like an appropriate way to talk about death march projects, tbh
Post reply on HN