Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jemalloc Postmortem

jasone.github.io

Post reply on HN
Print view View original

211–220 of 250 posts

Re: Jemalloc Postmortem

#211

Post by einpoklum »

username223 wrote: post #206
CamouflagedKiwi wrote: post #187

Earlier quoted context omitted.

Because you have to build it. If they don't use the same build system as you, you either want to invoke their system, or import it into yours. The former is unappealing if it's 'heavy' or doesn't play well as a subprocess; the latter can take a lot of time if the build process you're replicating is complex. I've done both before, and seen libraries at various levels of complexity; there is definitely a point where yo…

This. When step one is "install our weird build system," I'll immediately look for something else that meets my needs. All build systems suck, so everyone thinks they can write a better one, and too many people try. Pretty soon you end up having to learn a majority of this ( https://en.wikipedia.org/wiki/List_of_build_automation_softw... ) to get your code to compile.

If TCMalloc uses bazel, then you build it with Bazel. It just needs to install itself where you tell it to, and then either it has given you a pkg-config file, or otherwise, your own build system needs some library-finding logic for it ("find module" in CMake terms). Or - are you saying the problem is that you need to install Bazel?

Re: Jemalloc Postmortem

#212

Post by [deleted] »

liuliu wrote: post #183
schrep wrote: post #74

Your work was so impactful over a long period from Firefox to Facebook. Honored to have been a small part of it.

Your leadership on continuing investing in core technologies in Facebook were as fruitful as it could ever being. GraphQL, PyTorch, React to name a few cannot happen without.

[deleted]

Re: Jemalloc Postmortem

#213

Post by dao- »

liuliu wrote: post #183
schrep wrote: post #74

Your work was so impactful over a long period from Firefox to Facebook. Honored to have been a small part of it.

Your leadership on continuing investing in core technologies in Facebook were as fruitful as it could ever being. GraphQL, PyTorch, React to name a few cannot happen without.

Hmm, if I had to choose between not having Facebook and having React, I'd pick the former in a heartbeat. Not that this was a real choice, but it was nonetheless bitter to see colleagues join the behemoth that was Facebook.

Re: Jemalloc Postmortem

#214

Post by carey »

int_19h wrote: post #205
Iwan-Zotow wrote: post #188

Earlier quoted context omitted.

+1 windows def allocator is pos. Jemalloc rules

> windows def allocator Which one of them? These days it could mean HeapAlloc, or it could mean malloc from uCRT.

malloc in uCRT just calls HeapAlloc, though? You can see the code in ucrt\heap\malloc_base.cpp if you have the Windows SDK installed.

Programs can opt in to the _segment_ heap in their manifest, but it’s not necessarily any faster.

Re: Jemalloc Postmortem

#215

Post by bialpio »

ahartmetz wrote: post #202
stick_figure wrote: post #198

Earlier quoted context omitted.

This is actually tracked at a publicly visible URL: https://commondatastorage.googleapis.com/chromium-browser-cl... And the include graph analysis: https://commondatastorage.googleapis.com/chromium-browser-cl... The annotated red dots correspond to the last time Chrome developers did a big push to prune the include graph to optimize build time. It was effective, but there was push back. C++ developers just want magic…

Good that it's being tracked, but Jesus, these numbers! 110 CPU hours for a build. (Fortunately, it seems to be a little over half that for my CPU. "Cloud CPUs" are kinda slow.) I picked the 5001st largest file with includes. It's zoom_view_controller.cc, 140 lines in the .cc file, size with includes: 19.5 MB. Initially I picked the 5000th largest file with includes, but for devtools_target_ui.cc, I see a bit more le…

> I picked the 5001st largest file with includes. It's zoom_view_controller.cc, 140 lines in the .cc file, size with includes: 19.5 MB.

> Initially I picked the 5000th largest file with includes, but for devtools_target_ui.cc, I see a bit more legitimacy for having lots of includes. It has 384 "own" lines in he .cc file and, of course, also about 19.5 MB size with includes.

> A C++20 source file including some standard library headers easily bloats to a little under 1 MB IIRC, and that's already kind of unreasonable. 20x of that is very unreasonable.

I think you're not arguing pro-forward-declarations vs anti-forward-declarations here though - it sounds more like an argument for more granular header/source files? In .cc file, each and every include should be necessary for the file to compile (although looking at your example, bind.h seems to be unused and could be removed - looks like the file was refactored and the includes weren't cleaned up).

With that said, in the corresponding zoom_view_controller.h, the tab_interface.h include looks to be unnecessary so you did find one good example. :)

Re: Jemalloc Postmortem

#216

Post by mort96 »

prpl wrote: post #199
mort96 wrote: post #110

Earlier quoted context omitted.

They got downvoted because 1) comments of the form "I gave a chat bot a toy example of a task and it managed it" are tired and uninformative, and 2) because nobody was talking about anything which would make translating a Makefile into Bazel somehow relevant, nobody here has a Makefile which we wish was Bazel, we wish Google code was easier to work with

People are discussing things that are tedious work. I think the conversion to Bazel from a makefile is much more tedious and error prone than the reverse, in part because of Bazel sandboxing although that shouldn’t make much of a difference for a well-defined collection of Makefiles of a C library. The reverse should be much easier, which was the point of the post. Pointing it out as a capability (translation of buil…

What's really tedious is the constant chat bot spam.

Re: Jemalloc Postmortem

#217

Post by ahartmetz »

bialpio wrote: post #215
ahartmetz wrote: post #202

Earlier quoted context omitted.

Good that it's being tracked, but Jesus, these numbers! 110 CPU hours for a build. (Fortunately, it seems to be a little over half that for my CPU. "Cloud CPUs" are kinda slow.) I picked the 5001st largest file with includes. It's zoom_view_controller.cc, 140 lines in the .cc file, size with includes: 19.5 MB. Initially I picked the 5000th largest file with includes, but for devtools_target_ui.cc, I see a bit more le…

> I picked the 5001st largest file with includes. It's zoom_view_controller.cc, 140 lines in the .cc file, size with includes: 19.5 MB. > Initially I picked the 5000th largest file with includes, but for devtools_target_ui.cc, I see a bit more legitimacy for having lots of includes. It has 384 "own" lines in he .cc file and, of course, also about 19.5 MB size with includes. > A C++20 source file including some standa…

Yes, sure! I am arguing for whatever is necessary to reduce the total compilation cost. Pruning headers, rearranging source code to have fewer trivial modules and to reduce the size of very often included headers, even gasp sometimes using pointers just to reduce compile time! I understand that runtime performance is a very high priority for Blink, but it really doesn't matter sometimes if certain things are heap-allocated. Like things that are very expensive to instantiate anyway and that don't occur often. These will incidentally tend to have "heavy" headers, too.

Re: Jemalloc Postmortem

#218

Post by senderista »

Twirrim wrote: post #67
dpe82 wrote: post #5

Earlier quoted context omitted.

Why would they have to change? Sometimes software development is largely "done" and there isn't much more you need to do to a library.

While I certainly wish that more software would reach a "done" stage, I don't think jemalloc is necessarily there yet. Unfortunately I'm aware of there being bugs in the current version of jemalloc, when run in certain environment configurations, including memory leaks. I know the folks that found it were looking to report it, but I guess that won't happen now. Even from a quick look at the open issues, I can see htt…

jemalloc is used enough at Amazon that it would make sense for them to maintain it, but that's not really their style.

Re: Jemalloc Postmortem

#219

Post by senderista »

jeffbee wrote: post #7
dpe82 wrote: post #5

Earlier quoted context omitted.

Why would they have to change? Sometimes software development is largely "done" and there isn't much more you need to do to a library.

For an example of why an allocator is a maintenance treadmill, consider that C++ recently (relatively) added sized delete, and Linux recently gained transparent huge pages.

Another example is rseq (which was originally implemented for tcmalloc).

Re: Jemalloc Postmortem

#220

Post by senderista »

mrweasel wrote: post #139

Looking at all the comments and lightly browsing the source code, I'm amazed. Both at how much impact a memory allocator can make, but also how much code is involved. I'm not really sure what I expected, but somehow I expect a memory allocator to be ... smaller, simpler perhaps?

You can write a simple size-class allocator (even lock-free) in just a couple dozen lines of code. (I've done it both for interviews and for a work presentation.) But an allocator that is fast, scalable, and performs well over diverse workloads--that is HARD.
Post reply on HN