Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make Ubuntu packages 90% faster by rebuilding them

gist.github.com

Post reply on HN
Print view View original

221–230 of 375 posts

Re: Make Ubuntu packages 90% faster by rebuilding them

#221

Post by jcelerier »

vlovich123 wrote: post #144
smallstepforman wrote: post #139

Earlier quoted context omitted.

Agree for most short running apps. I updated my comment to reflect issues with apps that are constantly reallocating, and running for longer that 60 seconds. But you are absolutely correct for most short running apps, 99% recommended to replace glibc. However, there is an app or two where glibc stability doesnt trigger a pathological use cases, and you have no choice. Hence why its the default, since there are less c…

Looked at your updated post and it looks like you’re operating under wildly incorrect assumptions. 1. Fragmentation: MIMalloc and the newest TCMalloc definitely handle this better than glibc. This is well established in many many many benchmarks. 2. In terms of process lifetime, MIMalloc (Microsoft Cloud) and TCMalloc (Google Cloud) are designed to be run for massive long-lived services that continually allocate/deal…

I tried to use mimalloc and snmalloc and in both cases got crashes I don't get with glibc when interoperating with other libraries (libusb, jack, one that I suspect to be in the Nvidia driver) :(

Re: Make Ubuntu packages 90% faster by rebuilding them

#222

Post by Y_Y »

scns wrote: post #212
Y_Y wrote: post #202

Earlier quoted context omitted.

Challenge yourself to produce some numbers first. If there are many many many benchmarks it shouldn't be too difficult to link one. Just saying something is "well established" doesn't really help without some other context.

https://github.com/microsoft/mimalloc?tab=readme-ov-file#per...

Thanks for that. I have an innate skepticism of benchmarks from a company who wants to show their solution is best, and it seems their latest results are from 2021, but I couldn't find a better comparison myself either.

I do note that rpmalloc (old), Hermes (not public? doesn't compare with mimalloc) and also snmalloc (also Microsoft) have benchmarks of their own showing themselves to be best in some circumstances.

https://github.com/mjansson/rpmalloc-benchmark

https://arxiv.org/abs/2109.02922

https://github.com/SchrodingerZhu/bench_suite/blob/master/ou...

Re: Make Ubuntu packages 90% faster by rebuilding them

#224

Post by dietr1ch »

wengo314 wrote: post #217

"Make one Ubuntu package 90% faster by rebuilding it and switching the memory allocator" i wish i could slap people in the face over standard tcp/ip for clickbait. it was ONE package and some gains were not realized by recompilation. i have to give it to him, i have preloaded jemalloc to one program to swap malloc implementation and results have been very pleasant. not in terms of performance (did not measure) but in…

To me it's obviously a scam because there's no way such an improvement can be achieved globally with a single post explanation. 90% faster is a micro-benchmark number.

Re: Make Ubuntu packages 90% faster by rebuilding them

#225

Post by wiredfool »

emmelaich wrote: post #32

I think parsing once into a faster format (sqlite3 or parquet) would be more beneficial. https://simdjson.org/

Yeah, any of GeoPackage (sqllite db with spatial extensions), FlatGeoBuf, or GeoArrow/GeoParquet would be far better for querying.

Re: Make Ubuntu packages 90% faster by rebuilding them

#226

Post by _joel »

toyg wrote: post #198
vasco wrote: post #166

Earlier quoted context omitted.

Never seen the HN version of the 'install gentoo' meme before, more sophisticated definitely. > The goal of Gentoo is to have an operating system that builds all programs from source, instead of having pre-built binary packages. While this does allow for advanced speed and customizability, it means that even the most basic components such as the kernel must be compiled from source. It is known through out the Linux c…

Where does that blurb come from, chatgpt? I don't think it's true anymore, last time I checked I think Gentoo had a "normal" liveCD installation for the base system, which you could then recompile on your own if wanted.

GRP (Gentoo packages) existed at least 20 years ago, from my memory, as that's the last time I really used it in anger. I remeber packages being available and not having to rice everything, for sure.

Re: Make Ubuntu packages 90% faster by rebuilding them

#227

Post by eru »

darkwater wrote: post #182
eru wrote: post #163

Earlier quoted context omitted.

Nice, I'll try them out! LTO is great, but I have my doubts about -O3 (vs the more conservative -O2). UPDATE: bah, ALHP repos don't support the nvidia drivers. And I don't want to muck around with setting everything up again. Another update: I moved to nvidia-open, so now I can try the suggested repos.

All in under 2 hours. You were totally nerd-sniped.

Well, I moved to the new repos. But I had to change to the dkms drivers.

Seems to work so far, but nothing 'feels' faster either.

Re: Make Ubuntu packages 90% faster by rebuilding them

#228

Post by eru »

globular-toast wrote: post #178
eru wrote: post #57

Earlier quoted context omitted.

I used Gentoo for a while, but the temptation to endlessly fiddle with everything always let me to eventually break the system. (It's not Gentoo's fault, it's mine.) Afterwards I moved to ArchLinux, and that has been mostly fine for me. If you are using a fairly standard processor, then Gentoo shouldn't give you that much of an advantage?

I've broken Arch but never broken Gentoo. I think this more due to the fact I ran Arch first and you then Gentoo first, rather than any real difference between them. Gentoo is more stable than Arch by default, though. It's not actually a bleeding edge distro, but you can choose to run it that way if you wish. Gentoo is about choice.

> I think this more due to the fact I ran Arch first and you then Gentoo first, rather than any real difference between them.

I can believe that.

> Gentoo is more stable than Arch by default, though. It's not actually a bleeding edge distro, but you can choose to run it that way if you wish. Gentoo is about choice.

I actually had way more trouble with stuff breaking with Ubuntu. That's because every six months, when I did the distro upgrade, lots of stuff broke at once and it was hard to do root cause analysis.

With a rolling distribution, it's usually only one thing breaking at a time.

Re: Make Ubuntu packages 90% faster by rebuilding them

#229

Post by alextingle »

1dom wrote: post #213
smallstepforman wrote: post #127

Engineering is a compromise. The article shows most gains come from specialising the memory allocater. The thing to remember is that some projects are multithreaded, and allocate in one thread, use data in another and maybe deallocate in a 3rd. The allocator needs to handle this. So a speedup for one project may be a crash in another. Also, what about reallocation strategy? Some programs preallocate and never touch m…

Sorry, I'm struggling to make sense of this comment. I don't know C or C compilers very well at all, but I read the full gist and felt I learned a bunch of stuff and got a lot of value from it. But then when I read this top comment, it makes me concerned I've completely misunderstood the article. From the tone of this comment, I assume that I shouldn't ever do what's talked about in this gist and it's a terrible sugg…

I'll have a go at explaining: The process described in the article isn't a simple recipe that you can apply to any program to achieve similar results.

`jq` is a command-line program that fires up to do one job, and then dies. For such a program, the only property we really want to optimise is execution speed. We don't care about memory leaks, or how much memory the process uses (within reason). `jq` could probably avoid freeing memory completely, and that would be fine. So using a super-stupid allocator is a big win for `jq`. You could probably write your own and make it run even faster.

But for a program with different runtime characteristics, the results might be radically different. A long-lived server program might need to avoid memory bloat more than it needs to run fast. Or it might need to ensure stability more than speed or size. Or maybe speed does matter, but it's throughput speed rather than latency. Each of those cases need to be measured differently, and may respond better to different optimisation strategies.

The comment that confused you is just trying to speak a word of caution about applying the article's recipe in a simplistic way. In the real world, optimisation can be quite an involved job.

Re: Make Ubuntu packages 90% faster by rebuilding them

#230

Post by Tuna-Fish »

Y_Y wrote: post #222
scns wrote: post #212

Earlier quoted context omitted.

https://github.com/microsoft/mimalloc?tab=readme-ov-file#per...

Thanks for that. I have an innate skepticism of benchmarks from a company who wants to show their solution is best, and it seems their latest results are from 2021, but I couldn't find a better comparison myself either. I do note that rpmalloc (old), Hermes (not public? doesn't compare with mimalloc) and also snmalloc (also Microsoft) have benchmarks of their own showing themselves to be best in some circumstances. h…

More than that, based on the basic architecture of JEmalloc, mimalloc and tcmalloc, you should always expect their fragmentation behavior to be better for long-running software than the glibc malloc. (At the expense of consuming substantially more memory for very small programs with only a few allocations of a given size). The glibc malloc has nearly pessimal fragmentation behavior, you are very confused here.
Post reply on HN