Live data from Hacker News

Make Ubuntu packages 90% faster by rebuilding them

gist.github.com

361–370 of 375 posts

Re: Make Ubuntu packages 90% faster by rebuilding them

#361
I tried this with a 1.3GB file [1], and got Gemini to convert the jq query into a go program, that takes 6.6s the second time its run.(to cache the geojson file). My laptop isn't particularly fast (i7-1250p), and go's json handling isn't particularly fast either, so jq's time is not impressing me when run on a ryzen 9 desktop processor on a 500MB file.

It's surprising how quick this kind of processing can be in go.

[1] https://data.acgov.org/datasets/2b026350b5dd40b18ed7a321fdcd...

The program:

  package main

  import (
        "encoding/json"
        "fmt"
        "os"
  )

  type FeatureCollection struct {
        Features []Feature `json:"features"`
  }

  type Feature struct {
        Properties Properties `json:"properties"`
  }

  type Properties struct {
        TotalNetValue int    `json:"TotalNetValue"`
        SitusCity     string `json:"SitusCity"`
 }

  func main() {
        filename := "Parcels.geojson"

        data, err := os.ReadFile(filename)
        if err != nil {
                fmt.Println("Error reading file:", err)
                return
        }

        var featureCollection FeatureCollection
        err = json.Unmarshal(data, &featureCollection)
        if err != nil {
                fmt.Println("Error unmarshaling JSON:", err)
                return
        }

        for _, feature := range featureCollection.Features {
                if feature.Properties.TotalNetValue 

Re: Make Ubuntu packages 90% faster by rebuilding them

#362
post #70

Earlier quoted context omitted.

UBSAN is usually a debug build only thing. You can run it in production for some added safety, but it comes at a performance cost and theoretically, if you test all execution paths on a debug build and fix all complaints, there should be no benefit to running it in production.

I think it's time for the C/C++ communities to consider a mindset shift and pivot to having almost all protectors, canaries, sanitizers, assertions (e.g. via _GLIBCXX_ASSERTIONS) on by default and recommended for use in release builds in production. The opposite (i.e, the current state of affairs) should be discouraged and begrudginly accepted in select few cases. https://www.youtube.com/watch?v=gG4BJ23BFBE is a pres…

I do not think things like the time command need to be compiled with such things. It is pointless, but your suggestion here is to do it anyway. Why bother?

Assertions in release builds are a bad idea since they can be fairly expensive. It is a better idea to have a different variety of assertion like the verify statements that OpenZFS uses, which are assertions that run even in release builds. They are used in situations where it is extremely important for an assertion to be done at runtime, without the performance overhead of the less important assertions that are in performance critical paths.

Re: Make Ubuntu packages 90% faster by rebuilding them

#363
post #330
post #112

Earlier quoted context omitted.

Mario 64 gets flak for some files not being compiled with optimizations on. There is a YouTuber, whose name I forget at the moment, who has been doing optimization and improvements on it and constantly talks about how this is for the better due to the low instruction cache size and slow speed on the console. Compiling with optimizations explodes code size (unrolling links, inlining, etc) to the point it’s a net loss…

Kaze Emanuar[0] is the YouTuber you're thinking of. [0] https://www.youtube.com/@KazeN64

Yes thank you!

Re: Make Ubuntu packages 90% faster by rebuilding them

#364
post #362

Earlier quoted context omitted.

I think it's time for the C/C++ communities to consider a mindset shift and pivot to having almost all protectors, canaries, sanitizers, assertions (e.g. via _GLIBCXX_ASSERTIONS) on by default and recommended for use in release builds in production. The opposite (i.e, the current state of affairs) should be discouraged and begrudginly accepted in select few cases. https://www.youtube.com/watch?v=gG4BJ23BFBE is a pres…

I do not think things like the time command need to be compiled with such things. It is pointless, but your suggestion here is to do it anyway. Why bother? Assertions in release builds are a bad idea since they can be fairly expensive. It is a better idea to have a different variety of assertion like the verify statements that OpenZFS uses, which are assertions that run even in release builds. They are used in situat…

Why would I want potentially undefined behaviour in 'time'? I expect it to crash anytime it's about to enter UB. Sure, you may want to minimize such statements between the start/stop of the timer, but I expect any processing of stdout/stderr of the child process to be UB-proofed as much as possible.

I think it's a philosophical difference of opinions and it's one of the things that drive Rust, Go, C# etc. ahead - not merely language ergonomics (I hope Zig ends up as the language that replaces C). The society at large is mostly happy to take a 1-3% perf hit to get rid of buffer overflows and other UB-inducing errors.

But I agree with you on not having "expensive" asserts in releases.

Re: Make Ubuntu packages 90% faster by rebuilding them

#365
post #358

Earlier quoted context omitted.

it’s a total distortion of what the phrase means. Security through obscurity is when you run your sshd server on port 1337 instead of 22 without actually securing the server settings down, because you don’t think the hackers know how to portscan that high. Everyone runs on 22, but you obscurely run it elsewhere. “Nobody will think to look.” ASLR is nothing like that. It’s not that nobody thinks to look, it’s that the…

With attacks such as AnC, your logic fails. They can figure out the locations and get plenty of stable gadgets. Any shuffling of a deck of cards by Alice is pointless if Bob can inspect the deck after she shuffles them. It makes ASLR not very different from changing your sshd port. In both cases, this describes the security: https://web.archive.org/web/20240123122515if_/https://www.sy...

okay, sure, ASLR can be defeated by hardware leaks. The first rowhammer papers were over ten years ago, it's very old news. It's totally irrelevant to this thread. The fact that there exist designs that have hardware flaws which make them incapable of hosting a secure PRNG does not have any relevance to a discussion about the merits or lack thereof of a PRNG-based security measures. The systems you're referring to don't have secure PRNGs.

Words have meaning, god damn it! ASLR is not security through obscurity.

Edit: I was operating under the assumption that “AnC” was some new hotness, but no, this is the same stuff that’s always been around, timing attacks on the caches. And there’s still the same solution as there was back then: you wipe the caches out so your adversaries have no opportunity to measure the latencies. It’s what they always should have done on consumer devices running untrusted code.

Re: Make Ubuntu packages 90% faster by rebuilding them

#366
Isn't it against the rules to post a link like this where the content is walled behind needing an account to access?

I know most of us have accounts here, but when we're seeing something we want to read its not conducive to have to have to deal with roadblocks before we can see what is being talked about. Same goes for paywalls.

Re: Make Ubuntu packages 90% faster by rebuilding them

#367
post #286

Earlier quoted context omitted.

Vector operations like AVX512 will not magically make common software faster. The number of applications that deal with regular operations on large blocks of data is pretty much limited to graphical applications, neural networks and bulk cryptographic operations. Even audio processing doesn't benefit that much from vector operations because a codec's variable-size packets do not allow for efficient vectorization (the…

Vector operations are widely used in common software. Java uses AVX512 for sorting. glibc uses SIMD instructions for string operations.

Thanks for the correction. I hadn't considered bulk memory operations to be part of SIMD operation but it makes sense -- they operate on a larger grain than word-size so they can do the same operation with less micro-ops overhead.

Re: Make Ubuntu packages 90% faster by rebuilding them

#368
post #360

Earlier quoted context omitted.

I think it’s security through obscurity because in ASLR the randomized base address is a protected secret key material whereas in the ipv6 case it’s unprotected key material (eg every hop between two communicating parties sees the secret). It’s close though which is why IPv6 mapping efforts are much more heuristics based than ipv4 which you can just brute force (along with port #) quickly these days.

If you can look up the base address via AnC, is considering it to be a protected key material really correct?

I think that's why the threat model matters. I consider my SSH keys secure as long as they don't leave the local machine in plaintext form. However if the scenario changes to become "the adversary has arbitrary read access to your RAM" then that's obviously not going to work anymore.

Re: Make Ubuntu packages 90% faster by rebuilding them

#369

Or, if you want to do it pain free, install guix on it.

How would you do the things mentioned in the article using Guix? You could write your own custom package definitions, extending the default to change up compile flags and allocators, but then you need to do this for every single package (and maintain them all). I'm not sure Guix gives you much here, though maybe that's fine for one or two packages. The most pain-free option I can think of is the --tune flag (which is…

    guix build --with-configure-flag="jq=CFLAGS=-O3" jq
If you want it to be permanent, then you can use a guix home profile (that's a declarative configuration of your home directory) with a patch function in the package list there:

    (define llama-tune
      (options->transformation `((tune . "znver3")))) ; Zen 3

    (home-environment
     (packages (list (llama-tune (specification->package "llama")))))
or:

    (define jq-patch
      (options->transformation `((with-configure-flag . "jq=CFLAGS=-O3"))))
[...]

    (jq-patch (specification->package "jq"))
[...]

You can also write a 10 line guile script to automatically do it for all dependencies (I sometimes do--for example for emacs). That would cause a massive rebuild, though.

>The most pain-free option I can think of is the --tune flag (which is similar to applying -march=native), but

> packages have to be defined as tunable for it to work (and not many are).

We did it that way on purpose--from prior experience, otherwise, you would get a combinatorial explosion of different package combinations.

If it does help for some package X, please email us a 2 line patch adding (tunable? . #t) to that one package.

If you do use --tune, it will tune everything that is tuneable in the dependency graph. But at least all dependents (not dependencies) will be just grafted--not be rebuilt.

Re: Make Ubuntu packages 90% faster by rebuilding them

#370
post #170

Earlier quoted context omitted.

they don't but that defeats the purpose. i want to be able to inspect the insides, so binary formats are out, i have to easily open & read the content

I don't get how you can't put the content within the merged file and query it but good luck.

because that d be a several hundreds megabytes json :) Anyway with suggestion in another comment, i moved the root directory into an antivirus-excluded location, which gave me almost a 10 times performance win
Post reply on HN