Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make Ubuntu packages 90% faster by rebuilding them

gist.github.com

Post reply on HN
Print view View original

51–60 of 375 posts

Re: Make Ubuntu packages 90% faster by rebuilding them

#51

Post by topspin »

rlpb wrote: post #38

Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…

This is certainly true. Also, by replacing the allocator and changing compiler flags, you're possibly immunizing yourself from attacks that rely on some specific memory layout.

Re: Make Ubuntu packages 90% faster by rebuilding them

#52

Post by AceJohnny2 »

taeric wrote: post #16
jeffbee wrote: post #7

Earlier quoted context omitted.

> Are the packages all debug symbols? Upstream is stripped actually. My final build here is 43x larger than the distro binary, partly due to having plentiful debug info.

Sorry, I didn't mean symbols, I meant optimization level. Hilarious that they have symbols stripped, but still outclassed. Love to see why they wouldn't have more optimizations. I'm assuming this isn't the case for more packages? Feels too good.

> Hilarious that they have symbols stripped, but still outclassed.

Debug symbols have 0 runtime penalty, just storage. They're just another section of the binary, referenced by debuggers, and which the loader skips.

In any case, all distros break out the symbols into separate files so that they can have their (storage) cake and (debug) eat it too.

Re: Make Ubuntu packages 90% faster by rebuilding them

#53

Post by internetter »

topspin wrote: post #51
rlpb wrote: post #38

Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…

This is certainly true. Also, by replacing the allocator and changing compiler flags, you're possibly immunizing yourself from attacks that rely on some specific memory layout.

That is, if you are a believer in security via obscurity

Re: Make Ubuntu packages 90% faster by rebuilding them

#54

Post by frontfor »

internetter wrote: post #53
topspin wrote: post #51

Earlier quoted context omitted.

This is certainly true. Also, by replacing the allocator and changing compiler flags, you're possibly immunizing yourself from attacks that rely on some specific memory layout.

That is, if you are a believer in security via obscurity

Are you arguing that ASLR is “security via obscurity”?

Re: Make Ubuntu packages 90% faster by rebuilding them

#55

Post by kazinator »

kazinator wrote: post #50
taeric wrote: post #3

Surprised it would be 90%. Are the packages all debug symbols? I recall trying with march=native and seeing some improvement, but not enough to care at a system level.

Using preload on the stock Ubuntu binary, to give it mimalloc, they got a 44% speedup. By rebuilding the binary with different compiler options, but not changing malloc, they got an 20% speedup. If we naively multiply these speedups, we get 1.78: 78% faster. How it goes to 1.9 is that when you speed up only the program, but leave malloc the same, malloc matters to its performance a lot more. When the faster malloc is…

This is the same reason why a faster garbage collector makes a huge difference to compiled code compared to interpreted.

When an AST interpreted languages gets a VM, gets native code, each step reveals GC to be slow.

You might go from 1% time spent in the GC to 15% to 60% (numbers of out thin air).

Re: Make Ubuntu packages 90% faster by rebuilding them

#56

Post by wongarsu »

taeric wrote: post #16
jeffbee wrote: post #7

Earlier quoted context omitted.

> Are the packages all debug symbols? Upstream is stripped actually. My final build here is 43x larger than the distro binary, partly due to having plentiful debug info.

Sorry, I didn't mean symbols, I meant optimization level. Hilarious that they have symbols stripped, but still outclassed. Love to see why they wouldn't have more optimizations. I'm assuming this isn't the case for more packages? Feels too good.

Ubuntu is still building for CPUs with the x86-64-v1 feature set. On experimental images built for x86-64-v3 (roughly the feature set of 10 year old CPUs, a bit newer on low-power CPUs) they have observed 60% improvement on some benchmarks (and much less in some others). Some distros have switched to -v3, Ubuntu is still holding out to support older hardware. The author is compiling to the actual feature set of their specific CPU, which is nearly guaranteed to be even more than -v3.

Another thing the article adds is LTO, which in my experience also makes a huge difference: it makes your software a lot faster in certain cases, but also makes build times a lot worse. Spending a bit more time in the build process should be an easy call, but might be harder at the scale of a distro.

Re: Make Ubuntu packages 90% faster by rebuilding them

#57

Post by eru »

teitoklien wrote: post #40

Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase. After initial setup, it’s pretty simple and easy to use, I remember making a ton of friends at matrix’s Gentoo Linux channel, was fun times. https://www.gentoo.org/ Fun fact, initial ChromeOS was basically just custom Gentoo Linux install, I’m not sure if they still use Gentoo Lin…

I used Gentoo for a while, but the temptation to endlessly fiddle with everything always let me to eventually break the system. (It's not Gentoo's fault, it's mine.)

Afterwards I moved to ArchLinux, and that has been mostly fine for me.

If you are using a fairly standard processor, then Gentoo shouldn't give you that much of an advantage?

Re: Make Ubuntu packages 90% faster by rebuilding them

#58

Post by DrNosferatu »

Sometimes comes to mind:

Can’t one recompile the same exact Ubuntu packages you already have on your system with optimal flags for your specific hardware?

Plus, the configuration could be automated - just push a button to (lazy, in background) recompile.

Priority for the ones profiled to be performance critical.

Re: Make Ubuntu packages 90% faster by rebuilding them

#59

Post by topspin »

frontfor wrote: post #54
internetter wrote: post #53

Earlier quoted context omitted.

That is, if you are a believer in security via obscurity

Are you arguing that ASLR is “security via obscurity”?

I would, and there is no shame in it, as far as I'm concerned.

I don't need to outrun the bear. I just need to outrun you.

Re: Make Ubuntu packages 90% faster by rebuilding them

#60

Post by smarx007 »

topspin wrote: post #51
rlpb wrote: post #38

Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…

This is certainly true. Also, by replacing the allocator and changing compiler flags, you're possibly immunizing yourself from attacks that rely on some specific memory layout.

There are also many flags that should be enabled by default for non-debug builds like ubsan, stack protection, see https://news.ycombinator.com/item?id=35758898
Post reply on HN