Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Jemalloc Postmortem

jasone.github.io

Post reply on HN
Print view View original

201–210 of 250 posts

Re: Jemalloc Postmortem

#201

Post by bialpio »

ahartmetz wrote: post #194
bialpio wrote: post #193

Earlier quoted context omitted.

Chromium deviates from Google-wide policy and allows forward-declarations: https://chromium.googlesource.com/chromium/src/+/main/styleg... , "Forward declarations vs. #includes".

That is really nice to hear, but AFAICS it only means that it may change in the future. Because in current code, it was ~all includes last time I checked. Well, I remember one - very biased - example where I had a look at a class that was especially expensive to compile, like 40 seconds (on a Ryzen 7950X) and maybe 2 GB of RAM. It had under 200 LOC and didn't seem to do anything that's typically expensive to compile.…

> Because in current code, it was ~all includes last time I checked.

That's another matter - just because forward-declares are allowed, doesn't mean they are mandated, but in my experience the reviewers were paying attention to that pretty well.

Counter-exeamples to "~all includes": https://source.chromium.org/chromium/chromium/src/+/main:thi..., https://source.chromium.org/chromium/chromium/src/+/main:thi..., https://source.chromium.org/chromium/chromium/src/+/main:thi....

I picked couple random headers from the directory where I've contributed the most to blink, and from what I'm seeing, most of the classes that could be forward-declared, were. I have not looked at .cc files given that those tend to need to see the declaration (except when it's unused, but then why have a forward-decl at all?) or the compiler would complain about access into incomplete type.

> Well, I remember one - very biased - example where I had a look at a class that was especially expensive to compile, like 40 seconds (on a Ryzen 7950X) and maybe 2 GB of RAM. It had under 200 LOC and didn't seem to do anything that's typically expensive to compile... except for the stuff it included.

Maybe the stuff was actually being compiled because of some member in a class (so it was actually expensive to compile). Or maybe you stumbled upon a place where folks weren't paying attention. Hard to say without a concrete example. The "compile firewall" was added pretty recently I think, but I don't know if it's going to block anything from landing.

Edit: formatting (switched bulleted list into comma-separated because clearly I don't know how to format it).

Re: Jemalloc Postmortem

#202

Post by ahartmetz »

stick_figure wrote: post #198
ahartmetz wrote: post #194

Earlier quoted context omitted.

That is really nice to hear, but AFAICS it only means that it may change in the future. Because in current code, it was ~all includes last time I checked. Well, I remember one - very biased - example where I had a look at a class that was especially expensive to compile, like 40 seconds (on a Ryzen 7950X) and maybe 2 GB of RAM. It had under 200 LOC and didn't seem to do anything that's typically expensive to compile.…

This is actually tracked at a publicly visible URL: https://commondatastorage.googleapis.com/chromium-browser-cl... And the include graph analysis: https://commondatastorage.googleapis.com/chromium-browser-cl... The annotated red dots correspond to the last time Chrome developers did a big push to prune the include graph to optimize build time. It was effective, but there was push back. C++ developers just want magic…

Good that it's being tracked, but Jesus, these numbers!

110 CPU hours for a build. (Fortunately, it seems to be a little over half that for my CPU. "Cloud CPUs" are kinda slow.)

I picked the 5001st largest file with includes. It's zoom_view_controller.cc, 140 lines in the .cc file, size with includes: 19.5 MB.

Initially I picked the 5000th largest file with includes, but for devtools_target_ui.cc, I see a bit more legitimacy for having lots of includes. It has 384 "own" lines in he .cc file and, of course, also about 19.5 MB size with includes.

A C++20 source file including some standard library headers easily bloats to a little under 1 MB IIRC, and that's already kind of unreasonable. 20x of that is very unreasonable.

I don't think that I need to tell anyone on the Chrome team how to improve performance in software: you measure and then you grab the dumb low-hanging fruit first. From these results, it doesn't seem like anyone is working with the actual goal to improve the situation as long as the guidelines are followed on paper.

Re: Jemalloc Postmortem

#203

Post by crest »

boulos wrote: post #54
matoro wrote: post #48

Earlier quoted context omitted.

That was me that filed the Itanium test suite failure. :)

The Itanic was kind of great :). I'm convinced it helped sink SGI.

Itanium did its most most important job: it killed everything but ARM and POWER.

Re: Jemalloc Postmortem

#204

Post by anonymoushn »

klabb3 wrote: post #147
Svetlitski wrote: post #36

I understand the decision to archive the upstream repo; as of when I left Meta, we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file (my favorite was the time someone filed an issue because our test suite didn’t pass on Itanium lol). Still, it makes me sad to see. Jemalloc is still IMO the best-performing general-purpose malloc implementation that’s…

> we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file Why not? I mean this is complete drive-by comment, so please correct me, but there was a fully staffed team at Meta that maintained it, but was not in the best place to manage the issues?

Well, to be blunt, the company does not care about this, so it does not get done.

Re: Jemalloc Postmortem

#205

Post by int_19h »

Iwan-Zotow wrote: post #188
dazzawazza wrote: post #127

I've used jemalloc in every game engine I've written for years. It's just the thing to do. WAY faster on win32 than the default allocator. It's also nice to have the same allocator across all platforms. I learned of it from it's integration in FreeBSD and never looked back. jemalloc has help entertained a lot of people :)

+1 windows def allocator is pos. Jemalloc rules

> windows def allocator

Which one of them? These days it could mean HeapAlloc, or it could mean malloc from uCRT.

Re: Jemalloc Postmortem

#206

Post by username223 »

CamouflagedKiwi wrote: post #187
einpoklum wrote: post #99

Earlier quoted context omitted.

> TCMalloc is great, but is an absolute nightmare to use if you’re not using bazel custom-malloc-newbie question: Why is the choice of build system (generator) significant when evaluating the usability of a library?

Because you have to build it. If they don't use the same build system as you, you either want to invoke their system, or import it into yours. The former is unappealing if it's 'heavy' or doesn't play well as a subprocess; the latter can take a lot of time if the build process you're replicating is complex. I've done both before, and seen libraries at various levels of complexity; there is definitely a point where yo…

This. When step one is "install our weird build system," I'll immediately look for something else that meets my needs. All build systems suck, so everyone thinks they can write a better one, and too many people try. Pretty soon you end up having to learn a majority of this (https://en.wikipedia.org/wiki/List_of_build_automation_softw...) to get your code to compile.

Re: Jemalloc Postmortem

#208

Post by cogman10 »

apaprocki wrote: post #75
matoro wrote: post #48

Earlier quoted context omitted.

That was me that filed the Itanium test suite failure. :)

Ah, porting to HP Superdome servers. It’s like being handed a brochure describing the intricate details of the iceberg the ship you just boarded is about to hit in a few days. A fellow traveler, ahoy!

I worked on the Superdome servers back in the day. What a weird product. I still can't believe it was a profitable division (at my time circa 2011).

HP was going through some turbulent waters in those days.

Re: Jemalloc Postmortem

#209

Post by cogman10 »

boulos wrote: post #181
sitkack wrote: post #97

Earlier quoted context omitted.

Why was the sinking of SGI great?

Oh, that wasn't the intent. I meant two separate things. The Itanic itself was kind of fascinating, but mostly panned (hence the nickname). SGI's decision to built out Itanium systems may have helped precipitate their own downfall. That was sad.

Still makes me sad. I partially think a major reason for the demise was that it was simply constructed too soon. Compiler tech wasn't nearly good enough to handle the ISA.

Nowadays because of the efforts that have gone in to making SIMD effective, I'd think modern compilers would have an easier time taking advantage of that unique and strange uarch.

Re: Jemalloc Postmortem

#210

Post by kabdib »

matoro wrote: post #48
Svetlitski wrote: post #36

I understand the decision to archive the upstream repo; as of when I left Meta, we (i.e. the Jemalloc team) weren’t really in a great place to respond to all the random GitHub issues people would file (my favorite was the time someone filed an issue because our test suite didn’t pass on Itanium lol). Still, it makes me sad to see. Jemalloc is still IMO the best-performing general-purpose malloc implementation that’s…

That was me that filed the Itanium test suite failure. :)

one of the best books on Linux architecture i've read was the one on the Itanium port

i think, because Itanic broke a ton of assumptions

Post reply on HN