Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make Ubuntu packages 90% faster by rebuilding them

gist.github.com

Post reply on HN
Print view View original

331–340 of 375 posts

Re: Make Ubuntu packages 90% faster by rebuilding them

#331

Post by ryao »

dotancohen wrote: post #181
bigstrat2003 wrote: post #106

Earlier quoted context omitted.

No, that other definition is the incorrect one. Security by obscurity does not require that the attacker is ignorant of the fact you're using it. Say I have an IPv6 network with no firewall, simply relying on the difficulty of scanning the address space. I think that people would agree that I'm using security by obscurity, even if the attacker somehow found out I was doing this. The correct definition is simply "usin…

> The correct definition is simply "using obscurity as a security defense mechanism", nothing more. Also stated as "security happens in layers", and often obscurity is a very good layer for keeping most of the script kiddies away and keeping the logs clean. My personal favorite example is using a non-default SSH port. Even if you keep it under 1024, so it's still on a root-controlled port, you'll cut down the attacks…

I use non-standard SSH ports too. It does not improve theoretical security, but it does improve quality of life from generating smaller logs.

Re: Make Ubuntu packages 90% faster by rebuilding them

#332

Post by 0xbadcafebee »

kennysoona wrote: post #151
donio wrote: post #86

Earlier quoted context omitted.

> Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase. That's true but worth noting that "optimize" here doesn't necessarily refer to performance. I've been using Gentoo for 20 years and performance was never the reason. Gentoo is great if you know how you want things to work. Gentoo helps you get there.

If it wasn't for performance, what was gained in using it over something like Slackware and building only the packages you needed to?

> If it wasn't for performance, what was gained

A new hobby

Re: Make Ubuntu packages 90% faster by rebuilding them

#333

Post by vlovich123 »

MyOutfitIsVague wrote: post #307
vlovich123 wrote: post #141

Earlier quoted context omitted.

Are you sure about that? I’m not aware of any additional UB exploitation enabled by O3 vs O2 or Os.

Nope, I'm not sure about it. I remember when I was using Gentoo about 10 years ago, this was the common reason given for using -O2 instead of -O3 in your build flags, and I'm just speaking from that memory.

Yeah there’s lots of cargo culting FUD like that that happens (see other posters in this thread that claim that glibc is a more stable allocator )

Re: Make Ubuntu packages 90% faster by rebuilding them

#334

Post by vlovich123 »

fc417fc802 wrote: post #324
vlovich123 wrote: post #296

Earlier quoted context omitted.

I think it’s security through obscurity because in ASLR the randomized base address is a protected secret key material whereas in the ipv6 case it’s unprotected key material (eg every hop between two communicating parties sees the secret). It’s close though which is why IPv6 mapping efforts are much more heuristics based than ipv4 which you can just brute force (along with port #) quickly these days.

I'm finding this semantic rabbit hole surprisingly amusing. The problem with that line of reasoning is that it implies that data handling practices can determine whether or not a given scheme is security through obscurity. But that doesn't fit the prototypical example where someone uses a super secret and utterly broken home rolled "encryption" algorithm. Nor does it fit the example of someone being careless with the…

> The problem with that line of reasoning is that it implies that data handling practices can determine whether or not a given scheme is security through obscurity

Necessary but not sufficient condition. For example, if I’m transmitting secrets across the wire in plain text that’s clearly security through obscurity even if you’re relying on an otherwise secure algorithm. Security is a holistic practice and you can’t ignore secrets management separate from the algorithm blueprint (which itself is also a necessary but not sufficient condition).

Re: Make Ubuntu packages 90% faster by rebuilding them

#335

Post by wantless »

_ea1k wrote: post #299
teitoklien wrote: post #40

Gentoo linux is essentially made specifically for people like this, to be able to optimize one’s own linux rig for one’s specific usecase. After initial setup, it’s pretty simple and easy to use, I remember making a ton of friends at matrix’s Gentoo Linux channel, was fun times. https://www.gentoo.org/ Fun fact, initial ChromeOS was basically just custom Gentoo Linux install, I’m not sure if they still use Gentoo Lin…

I tried that once, but it is still compiling. :) Kidding... honestly that was a pretty fun distribution to play around with ~20 years ago. The documentation was really good and it was a great way to learn how a lot of the pieces of a Linux distribution fit together. I was never convinced that the performance difference was really noticeable, though.

Gentoo was the primary source of heating for my living quarters back in the early 2000s. My tower was highly constrained on memory, and I was on a relentless quest to pare out any modules or dependencies I wasn't actually using. Performance gains were primarily from being able to stay out of slow HDD swap space memory. I doubt there were any gains once amortizing the compilation times, but I ran my compile batches at night, and they kept me nice and warm.

Re: Make Ubuntu packages 90% faster by rebuilding them

#336

Post by rlpb »

taeric wrote: post #124
rlpb wrote: post #38

Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…

I'm curious how applicable these are, in general? Feels like pointing out that using interior doors in your house misses out on the security afforded from a vault door. Not wrong, but there is also a reason every door in a bank is not a vault door. That is, I don't want to devalue the CVE system; but it is also undeniable that there are major differences in impact between findings?

Sure, but jq is very much a "front door" in your analogy. You'd have to look at each individual CVE to assess the risk for your specific case, but for jq, claimed security vulnerabilities are worth paying attention to.

Re: Make Ubuntu packages 90% faster by rebuilding them

#337

Post by rlpb »

jeffbee wrote: post #93
rlpb wrote: post #38

Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…

This is generally true but specifically false. The builds described in the gist are still linking onigurama dynamically. It is in another package, libonig5, that would be updated normally.

The gist uses the orig tarball only, so skips the distro patch that selects the distro packaged libonig in favor of the "vendored" one. At least that's how it appears to me. I only skimmed the situation.

Or do you see something deeper that ensures that the distro libonig is actually the one that gets used?

Re: Make Ubuntu packages 90% faster by rebuilding them

#338

Post by UncleEntity »

tonymet wrote: post #312
wengo314 wrote: post #217

"Make one Ubuntu package 90% faster by rebuilding it and switching the memory allocator" i wish i could slap people in the face over standard tcp/ip for clickbait. it was ONE package and some gains were not realized by recompilation. i have to give it to him, i have preloaded jemalloc to one program to swap malloc implementation and results have been very pleasant. not in terms of performance (did not measure) but in…

the title is clickbait, but it's good to encourage app developers to rebuild. esp when you are cpu bound on a few common utitilities e.g. jq, grep, ffmpeg, ocrmypdf -- common unix utils built build targets for general use rather than a specific application

Or, if I understand TFA correctly, don't release debug builds in your release packages.

Reminds me of back in the day, when I was messing around with blender's cmake config files quite a bit, I noticed the fedora package was using the wrong flag -- some sort of debug only flag intended for developers instead of whatever they thought is was. I mentioned this to the package maintainer, it was confirmed by package sub-maintainer (or whomever) and the maintainer absolutely refused to change it because the spelling of the two flags was close enough they could just say "go away, contributing blender dev, you have no idea what you're talking about." Wouldn't doubt the fedora package still has the same mistaken flag to this day and all this occurred something like 15 years ago.

So, yeah, don't release debug builds if you're a distro package maintainer.

Re: Make Ubuntu packages 90% faster by rebuilding them

#339

Post by datacynic »

llimllib wrote: post #81
emmelaich wrote: post #32

I think parsing once into a faster format (sqlite3 or parquet) would be more beneficial. https://simdjson.org/

For kicks, I downloaded the file and compared duckdb with the spatial extension against jq. jq was about 2x as fast for interactive use, but if you had multiple queries to run, paying a small cost to create a duckdb database and then querying it would be vastly faster

I also was nerdsniped into trying this and found that after extracting the features array into a newline delimited json file, DuckDB finishes the example query in 500 ms (M1 Mac), querying the 1.3 GB json file directly with read_json!

Re: Make Ubuntu packages 90% faster by rebuilding them

#340

Post by kennysoona »

0xbadcafebee wrote: post #332
kennysoona wrote: post #151

Earlier quoted context omitted.

If it wasn't for performance, what was gained in using it over something like Slackware and building only the packages you needed to?

> If it wasn't for performance, what was gained A new hobby

That's very fair, I've certainly tinkered with stuff for the fun of it when there were easier or more suitable alternatives.
Post reply on HN