Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make Ubuntu packages 90% faster by rebuilding them

gist.github.com

Post reply on HN
Print view View original

131–140 of 375 posts

Re: Make Ubuntu packages 90% faster by rebuilding them

#131

Post by [deleted] »

vlovich123 wrote: post #129
smallstepforman wrote: post #127

Engineering is a compromise. The article shows most gains come from specialising the memory allocater. The thing to remember is that some projects are multithreaded, and allocate in one thread, use data in another and maybe deallocate in a 3rd. The allocator needs to handle this. So a speedup for one project may be a crash in another. Also, what about reallocation strategy? Some programs preallocate and never touch m…

Mimalloc is a general purpose allocator like JEMalloc / TCMalloc. Glibc is known to have a pretty bad allocator that modern allocators like MIMalloc & the latest TCMalloc (not the one available by default in Ubuntu) run laps around. While of course speedup may be variable, generally the benchmarks show an across the board speedup (whether that matters for any given application is something entirely different). As for…

[deleted]

Re: Make Ubuntu packages 90% faster by rebuilding them

#132

Post by graymatters »

ryao wrote: post #76
frontfor wrote: post #54

Earlier quoted context omitted.

Are you arguing that ASLR is “security via obscurity”?

ASLR is technically a form of security by obscurity. The obscurity here being the memory layout. The reason nobody treated it that way was the high entropy that ASLR had on 64-bit, but the ASLR⊕Cache attack has undermined that significantly. You really do not want ASLR to be what determines whether an attacker takes control of your machine if you care about having a secure system.

You are confusing randomization, a legitimate security mechanism, with security by obscurity. ASLR is not security by obscurity. Please spend the time on understanding the terminology rather than regurgitating buzz words.

Re: Make Ubuntu packages 90% faster by rebuilding them

#133

Post by vlovich123 »

bigstrat2003 wrote: post #106
minitech wrote: post #103

Earlier quoted context omitted.

I assume people downvoted it because “ASLR obscures the memory layout. That is security by obscurity by definition” is just wrong (correct description here: https://news.ycombinator.com/item?id=43408039 ). It does say [flagged] too, though, so maybe that’s not the whole story…?

No, that other definition is the incorrect one. Security by obscurity does not require that the attacker is ignorant of the fact you're using it. Say I have an IPv6 network with no firewall, simply relying on the difficulty of scanning the address space. I think that people would agree that I'm using security by obscurity, even if the attacker somehow found out I was doing this. The correct definition is simply "usin…

ASLR is not purely security through obscurity because it is based on a solid security principle: increasing the difficulty of an attack by introducing randomness. It doesn't solely rely on the secrecy of the implementation but rather the unpredictability of memory addresses.

Think of it this way - if I guess the ASLR address once, a restart of the process renders that knowledge irrelevant implicitly. If I get your IPv6 address once, you’re going to have to redo your network topology to rotate your secret IP. That’s the distinction from ASLR.

Re: Make Ubuntu packages 90% faster by rebuilding them

#135

Post by atonse »

behnamoh wrote: post #92
atonse wrote: post #26

I’m almost more amazed that someone figured out jq’s syntax and got some use out of it. In all seriousness though, are you sure some of this isn’t those blocks being loaded into some kind of file system cache the second and third times? How about if you rebooted and then ran the mimalloc version?

jq has point-free programming. it's intuitive once you wrap your head around it. See this: https://en.wikipedia.org/wiki/Tacit_programming#jq

Yeah after this post I sought a couple youtube videos explaining it. It's starting to make a bit more sense now. But the lightbulb hasn't gone off just yet. Appreciate the link.

Re: Make Ubuntu packages 90% faster by rebuilding them

#136

Post by pertymcpert »

taeric wrote: post #123
rgmerk wrote: post #111

It's a while since I had to deal with this kind of thing, but my memory was that as soon as you go beyond the flags that the upstream developers use (just to be clear, I mean the upstream developers, not the distro packagers) you're buying yourself weird bugs and a whole lot of indifference if they occur. I haven't used a non-libc malloc before but I suspect the same applies.

I've been building my own emacs for a long time, and have yet to hit any weird bugs. I thought that as long as you avoid any unsafe optimizations, you should be fine? Granted, I also thought that -march=native was the main boost that I was seeing. This post indicates that is not necessarily the case. I also suspect that any application using floats is more likely to have rough edges?

Complex software usually has some undefined behavior lurking that at higher or even just different optimization levels can trigger the compiler to do unexpected things to the code. It happens all the time in my line of work. If there's an extensive test suite you can run to verify that it still works mostly as expected then it's easier.

Re: Make Ubuntu packages 90% faster by rebuilding them

#137

Post by vlovich123 »

taeric wrote: post #116
wongarsu wrote: post #56

Earlier quoted context omitted.

Ubuntu is still building for CPUs with the x86-64-v1 feature set. On experimental images built for x86-64-v3 (roughly the feature set of 10 year old CPUs, a bit newer on low-power CPUs) they have observed 60% improvement on some benchmarks (and much less in some others). Some distros have switched to -v3, Ubuntu is still holding out to support older hardware. The author is compiling to the actual feature set of their…

I was under the, evidently naive, understanding that there were some shenanigans done in the libc to pick optimal code paths based on the architecture. This would obviously have a startup cost on the first run, but presumably the dynamic linking would work to this advantage by having it in a fast state for most calls? Not sure where I got that idea, though. :(. Will have to look into that later.

Even if libc does it, other packages wouldn’t. The compiler is what generates code and they generally don’t try to automatically generate for different instruction sets because they don’t know what code is what hotpath and worth optimizing for. You could probably try to build for multiple different CPU features at the top level and do a universal executable like Apple did for their CPU transitions but that’s a lot of expense to pay both in terms of compilation time AND in terms of size.

Re: Make Ubuntu packages 90% faster by rebuilding them

#138

Post by oguz-ismail »

electromech wrote: post #79

I'd be curious how the performance compares to this Rust jq clone: cargo install --locked jaq (you might also be able to add RUSTFLAGS="-C target-cpu=native" to enable optimizations for your specific CPU family) "cargo install" is an underrated feature of Rust for exactly the kind of use case described in the article. Because it builds the tools from source, you can opt into platform-specific features/instructions th…

> I'd be curious how the performance compares to this Rust jq clone

Every once in a while I test jaq against jq and gojq with my jq solution to AoC 2022 day 13 https://gist.github.com/oguz-ismail/8d0957dfeecc4f816ffee79d...

It's still behind both as of today

Re: Make Ubuntu packages 90% faster by rebuilding them

#139

Post by smallstepforman »

vlovich123 wrote: post #129
smallstepforman wrote: post #127

Engineering is a compromise. The article shows most gains come from specialising the memory allocater. The thing to remember is that some projects are multithreaded, and allocate in one thread, use data in another and maybe deallocate in a 3rd. The allocator needs to handle this. So a speedup for one project may be a crash in another. Also, what about reallocation strategy? Some programs preallocate and never touch m…

Mimalloc is a general purpose allocator like JEMalloc / TCMalloc. Glibc is known to have a pretty bad allocator that modern allocators like MIMalloc & the latest TCMalloc (not the one available by default in Ubuntu) run laps around. While of course speedup may be variable, generally the benchmarks show an across the board speedup (whether that matters for any given application is something entirely different). As for…

Agree for most short running apps. I updated my comment to reflect issues with apps that are constantly reallocating, and running for longer that 60 seconds. But you are absolutely correct for most short running apps, 99% recommended to replace glibc. However, there is an app or two where glibc stability doesnt trigger a pathological use cases, and you have no choice. Hence why its the default, since there are less crashes in pathaligical cases. And the devs are exhausted dealing with crashing bugs which can be eliminated by using the slower allocator.
Post reply on HN