Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make Ubuntu packages 90% faster by rebuilding them

gist.github.com

Post reply on HN
Print view View original

261–270 of 375 posts

Re: Make Ubuntu packages 90% faster by rebuilding them

#261

Post by cb321 »

stakhanov wrote: post #209
CamouflagedKiwi wrote: post #189

Earlier quoted context omitted.

USE flags. You can build packages with specific features enabled or disabled, which can further reduce your dependency tree.

Reducing the dependency tree gets a bit more complicated once you consider that now you have to satisfy not only runtime dependencies for all packages but also build-time dependencies. There may be ways of cleaning that up after a build, but next time you want to emerge a new package you'll just end up having to re-build the build-time dependencies, so in practice you'll just end up leaving them there. There is an ab…

This is a good point. I've been using Gentoo since early 2004 (the dreaded Pentium IV era, Lol). Lately, I run into this with dev-lang/tcl only being need to build dev-db/sqlite. I actually think it's pretty weird that software intended to be as widely used as sqlite with as much of a free base of supporting devs doesn't just do the extra effort to use a Makefile.

Re: Make Ubuntu packages 90% faster by rebuilding them

#262

Post by draw_down »

wengo314 wrote: post #217

"Make one Ubuntu package 90% faster by rebuilding it and switching the memory allocator" i wish i could slap people in the face over standard tcp/ip for clickbait. it was ONE package and some gains were not realized by recompilation. i have to give it to him, i have preloaded jemalloc to one program to swap malloc implementation and results have been very pleasant. not in terms of performance (did not measure) but in…

[dead]

Re: Make Ubuntu packages 90% faster by rebuilding them

#263

Post by wengo314 »

glandium wrote: post #255
FooBarWidget wrote: post #248

Earlier quoted context omitted.

I did research into the glibc memory allocator. Turns out this is not memory fragmentation, but per-thread caches that are never freed back to the kernel! A free() call does not actually free the memory externally unless in exceptional circumstances. The more threads and CPU cores you have, the worse this problem becomes. One easy solution is setting the "magic" environment variable MALLOC_ARENA_MAX=2, which limits t…

The glibc memory allocator DOES have pathological cases that lead to what can look like memory leaks. See https://glandium.org/blog/?p=3698 https://glandium.org/blog/?p=3723 https://sourceware.org/bugzilla/show_bug.cgi?id=23416 (despite being rather old, it's also still a problem)

FWIW i had it with icinga2. so now they actually preload jemalloc in the service file to mitigate the issue, this may very well be what you're talking about

in case someone is interested: https://github.com/Icinga/icinga2/issues/8737

(basically using jemalloc was the big fix)

https://icinga.com/docs/icinga-2/latest/doc/15-troubleshooti...

Re: Make Ubuntu packages 90% faster by rebuilding them

#264

Post by wengo314 »

looofooo0 wrote: post #251
dietr1ch wrote: post #224

Earlier quoted context omitted.

To me it's obviously a scam because there's no way such an improvement can be achieved globally with a single post explanation. 90% faster is a micro-benchmark number.

True that, I mean it is still interesting, that if you have a narrow task, you might achieve some significant speed up from rebuilding them. But this is a very niche application.

true, i saw a thread recently on reddit where guy hand-tuned compilation flags and did pgo profiling for a video encoder app that he uses on video encode farm.

In his case, even a gain of ~20% was significant. It calculated into extra bandwidth to encode a few thousand more video files per year.

Re: Make Ubuntu packages 90% faster by rebuilding them

#265

Post by blablabla123 »

wengo314 wrote: post #217

"Make one Ubuntu package 90% faster by rebuilding it and switching the memory allocator" i wish i could slap people in the face over standard tcp/ip for clickbait. it was ONE package and some gains were not realized by recompilation. i have to give it to him, i have preloaded jemalloc to one program to swap malloc implementation and results have been very pleasant. not in terms of performance (did not measure) but in…

True, I also believed it for a second. But it's also easy to blame Ubuntu for errors. IMHO they are doing a quite decent job with assembling their packages. In fact they are also compiled with Stack fortifications. On the other hand I'm glad they are not compiled with the possibly buggy -O3. It can be nice for something performance critical but I definitely don't want a whole system compiled with -O3.

Re: Make Ubuntu packages 90% faster by rebuilding them

#266

Post by tpoacher »

Obviously the title / article is clickbait.

But in general I have often found great benefit in many cases in obtaining packages from source and compiling them, or downloading official binaries, even if the repository does have the latest version in theory.

(shameless plug incoming): My only obstacle with such "Manually Installed [and/or] Source-Compiled" (MISC) packages was checking for updates. So I made this: https://sr.ht/~tpapastylianou/misc-updater/

Works great, and I've been slowly improving it to add 'upgrader' commands for some of the packages too.

It's not a full replacement for a package manager, nor is it meant to be. But it's made my ability to use MISC packages instead of repository ones a lot more streamlined and easier.

Re: Make Ubuntu packages 90% faster by rebuilding them

#267

Post by rurban »

dan-robertson wrote: post #31
kstrauser wrote: post #5

(Well, rebuilding them with a different allocator that benchmarks well on their specific workflow.)

Is it even known what workloads the glibc malloc is good for?

It's using the least additional memory, compared to all others. Whilst being the slowest by far of all modern allocators.

Re: Make Ubuntu packages 90% faster by rebuilding them

#268

Post by rurban »

chasil wrote: post #216
pests wrote: post #112

Earlier quoted context omitted.

Mario 64 gets flak for some files not being compiled with optimizations on. There is a YouTuber, whose name I forget at the moment, who has been doing optimization and improvements on it and constantly talks about how this is for the better due to the low instruction cache size and slow speed on the console. Compiling with optimizations explodes code size (unrolling links, inlining, etc) to the point it’s a net loss…

The -Os flag optimizes for size. What impact does that have?

The IC is the instruction cache. Bloated code needs more slow loads from the IC, and even if the performance is faster per cache line, the 50x overhead of added cache loads diminishes all -O3 advantages.

Re: Make Ubuntu packages 90% faster by rebuilding them

#269

Post by rurban »

roeles wrote: post #215
rurban wrote: post #190

He is still missing profile guided optimizations (with his testcase), and bolt. And in some/most cases -Os is faster than -O3 Boehm GC is usually slower than glibc with lots of allocs. MPS would be better, but needs lots of rewriting. https://github.com/Ravenbrook/mps

> And in some/most cases -Os is faster than -O3 This surprises me. Can you please elaborate?

A IC cache miss costs 50x more on average. If your insn and data are small, the overhead of doing more work is less than the overhead of cache misses

Re: Make Ubuntu packages 90% faster by rebuilding them

#270

Post by kllrnohj »

blablabla123 wrote: post #265
wengo314 wrote: post #217

"Make one Ubuntu package 90% faster by rebuilding it and switching the memory allocator" i wish i could slap people in the face over standard tcp/ip for clickbait. it was ONE package and some gains were not realized by recompilation. i have to give it to him, i have preloaded jemalloc to one program to swap malloc implementation and results have been very pleasant. not in terms of performance (did not measure) but in…

True, I also believed it for a second. But it's also easy to blame Ubuntu for errors. IMHO they are doing a quite decent job with assembling their packages. In fact they are also compiled with Stack fortifications. On the other hand I'm glad they are not compiled with the possibly buggy -O3 . It can be nice for something performance critical but I definitely don't want a whole system compiled with -O3.

> with the possibly buggy -O3

-O3 isn't buggy on either GCC or Clang. Are you thinking of -Ofast and/or -ffast-math that disregard standards compliance? Those aren't part of -O3.

Post reply on HN