Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make Ubuntu packages 90% faster by rebuilding them

gist.github.com

Post reply on HN
Print view View original

291–300 of 375 posts

Re: Make Ubuntu packages 90% faster by rebuilding them

#291

Post by blagie »

vlovich123 wrote: post #231
jcelerier wrote: post #221

Earlier quoted context omitted.

I tried to use mimalloc and snmalloc and in both cases got crashes I don't get with glibc when interoperating with other libraries (libusb, jack, one that I suspect to be in the Nvidia driver) :(

If you are not properly overriding the allocator consistently for everything within an executable, that’s entirely possible (eg linking against 1 allocator and then linking with a dynamic library that’s using a different one). Without a specific repro it’s hard to distinguish PEBCAK from legit bug. Also it certainly can’t be the Nvidia driver since that’s not running anything in your process.

The explanation here is usually simpler. Any major change like this can lead to crashes.

If I have three libraries, biz, bar, and bif, and each has bugs. You've used biz. When the system was unstable, you'd debug until it works (either by finding the real bug, or by an innocuous change).

If you switch libraries, the bugs which go away are the ones which aren't affecting you, since your software works. On the other hand, you have a whole new set of bugs.

This comes up when upgrading libraries too. More bugs are usually fixed than introduced, but there's often a debug cycle.

Re: Make Ubuntu packages 90% faster by rebuilding them

#292

Post by usefulcat »

bigstrat2003 wrote: post #105
usefulcat wrote: post #100

Earlier quoted context omitted.

I'm genuinely curious what was so undesirable about this sibling comment that it was removed: "ASLR obscures the memory layout. That is security by obscurity by definition. People thought this was okay if the entropy was high enough, but then the ASLR⊕Cache attack was published and now its usefulness is questionable." Usually when a comment is removed, it's pretty obvious why, but in this case I'm really not seeing i…

Some people use the "flag" button as a "disagree" button or even a "fuck this guy" button. Unfortunately, constructive but unpopular comments get flagged to death on HN all the time.

I had thought that flagging was basically a request for a mod to have a look at something. But based on this case I now suspect that it's possible for a comment to be removed without a mod ever looking at it if enough people flag it.

Re: Make Ubuntu packages 90% faster by rebuilding them

#293

Post by cratermoon »

wengo314 wrote: post #217

"Make one Ubuntu package 90% faster by rebuilding it and switching the memory allocator" i wish i could slap people in the face over standard tcp/ip for clickbait. it was ONE package and some gains were not realized by recompilation. i have to give it to him, i have preloaded jemalloc to one program to swap malloc implementation and results have been very pleasant. not in terms of performance (did not measure) but in…

I wonder how many prepackaged binary distributions are built with the safest options for the os/hardware and don't achieve the best possible performance. I bet most of them, tbh. Many years ago I started building Mozilla and my own linux kernels to my preferences, usually realizing modest performance gains. The entire purpose of the Gentoo Linux distribution, e.g., is performance gains possible by optimized compilation of everything from source.

Re: Make Ubuntu packages 90% faster by rebuilding them

#294

Post by throw16180339 »

usefulcat wrote: post #100
msm_ wrote: post #69

Earlier quoted context omitted.

ASLR is not security through obscurity though. It forces attacker to get a pointer leak before doing almost anything (even arbitrary read and arbitrary write primitives are useless without a leak with ASLR). As someone with a bit of experience in exploit dev, it makes a world of a difference and is one of the most influential hardenings, next to maybe stack cookies and W^X.

I'm genuinely curious what was so undesirable about this sibling comment that it was removed: "ASLR obscures the memory layout. That is security by obscurity by definition. People thought this was okay if the entropy was high enough, but then the ASLR⊕Cache attack was published and now its usefulness is questionable." Usually when a comment is removed, it's pretty obvious why, but in this case I'm really not seeing i…

I downvoted because the poster doesn't understand what security by obscurity means.

Re: Make Ubuntu packages 90% faster by rebuilding them

#295

Post by boobsbr »

teitoklien wrote: post #40

Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase. After initial setup, it’s pretty simple and easy to use, I remember making a ton of friends at matrix’s Gentoo Linux channel, was fun times. https://www.gentoo.org/ Fun fact, initial ChromeOS was basically just custom Gentoo Linux install, I’m not sure if they still use Gentoo Lin…

Obligatory "Gentoo is rice" mention:

https://www.shlomifish.org/humour/by-others/funroll-loops/Ge...

Re: Make Ubuntu packages 90% faster by rebuilding them

#296

Post by vlovich123 »

fc417fc802 wrote: post #276
vlovich123 wrote: post #237

Earlier quoted context omitted.

You don’t necessarily need to expose the IPv6 address to untrusted parties though in which case it is indeed quite similar to ASLR in that data leakage of some kind is necessary. I think the main distinguishing factor is that ASLR by design treats the base address as a secret and guards it as such whereas that’s not a mode the IPv6 address can have because by its nature it’s assumed to be something public.

Huh. The IPv6 example is much more confusing that I initially thought. At this point I am entirely unclear as to whether it is actually an example of security through obscurity, regardless of whatever else it might be (a very bad idea to rely on it for one). Rather ironic given that the poster whose claims I was disputing provided it as an example of something that would be universally recognized as such.

I think it’s security through obscurity because in ASLR the randomized base address is a protected secret key material whereas in the ipv6 case it’s unprotected key material (eg every hop between two communicating parties sees the secret). It’s close though which is why IPv6 mapping efforts are much more heuristics based than ipv4 which you can just brute force (along with port #) quickly these days.

Re: Make Ubuntu packages 90% faster by rebuilding them

#297

Post by hedora »

scottlamb wrote: post #278
smallstepforman wrote: post #127

Engineering is a compromise. The article shows most gains come from specialising the memory allocater. The thing to remember is that some projects are multithreaded, and allocate in one thread, use data in another and maybe deallocate in a 3rd. The allocator needs to handle this. So a speedup for one project may be a crash in another. Also, what about reallocation strategy? Some programs preallocate and never touch m…

> I experimented with different allocators devoloping a video editor testing 4K videos that caches frames. 32Mb per frame, at 60fps, thats almost 2Gb per second per track. You quickly hit allocator limitations, and realise that at least vanilla glibc allocator offers the best long term stability. But for short running benchmarks its the slowest. I also work with large (8K) video frames [1]. If you're talking about th…

I’d be shocked if jemalloc or tcmalloc had issues with that workload.

Do you have a minimal reproducing example?

Re: Make Ubuntu packages 90% faster by rebuilding them

#298

Post by buttercraft »

usefulcat wrote: post #292
bigstrat2003 wrote: post #105

Earlier quoted context omitted.

Some people use the "flag" button as a "disagree" button or even a "fuck this guy" button. Unfortunately, constructive but unpopular comments get flagged to death on HN all the time.

I had thought that flagging was basically a request for a mod to have a look at something. But based on this case I now suspect that it's possible for a comment to be removed without a mod ever looking at it if enough people flag it.

Not removed but hidden. You can turn on showdead in your profile to see them.

Re: Make Ubuntu packages 90% faster by rebuilding them

#299

Post by _ea1k »

teitoklien wrote: post #40

Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase. After initial setup, it’s pretty simple and easy to use, I remember making a ton of friends at matrix’s Gentoo Linux channel, was fun times. https://www.gentoo.org/ Fun fact, initial ChromeOS was basically just custom Gentoo Linux install, I’m not sure if they still use Gentoo Lin…

I tried that once, but it is still compiling. :)

Kidding... honestly that was a pretty fun distribution to play around with ~20 years ago. The documentation was really good and it was a great way to learn how a lot of the pieces of a Linux distribution fit together.

I was never convinced that the performance difference was really noticeable, though.

Re: Make Ubuntu packages 90% faster by rebuilding them

#300

Post by vlovich123 »

jeffbee wrote: post #283
scns wrote: post #212

Earlier quoted context omitted.

https://github.com/microsoft/mimalloc?tab=readme-ov-file#per...

Oddly enough I've written about those results before: https://www.dropbox.com/scl/fi/evnn6yoornh9p6l7nq1t9/Irrepro...

but we all can agree that glibc is trash right? :)

I’d be careful extrapolating from just one benchmark but generally if I had to choose I’d pick the new tcmalloc if I could. It seems to be a higher quality codebase.

Post reply on HN