Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Make Ubuntu packages 90% faster by rebuilding them

gist.github.com

Post reply on HN
Print view View original

91–100 of 375 posts

Re: Make Ubuntu packages 90% faster by rebuilding them

#91

Post by jeffbee »

electromech wrote: post #84
jeffbee wrote: post #21

Earlier quoted context omitted.

jaq is nice. It just loses, performance-wise, to the final step in this article, and it can't do the second mentioned workload (yet) so I didn't include it.

Which workload can't it do? I've had good success with jaq performance.

It bombs out on the jq program I use for the 2nd corpus that I mentioned. On further investigation, the show-stopping filter is strftime. In the jaq readme this is the only not-yet-checked box in the compatibility list, so perhaps some day soon.

Re: Make Ubuntu packages 90% faster by rebuilding them

#92

Post by behnamoh »

atonse wrote: post #26

I’m almost more amazed that someone figured out jq’s syntax and got some use out of it. In all seriousness though, are you sure some of this isn’t those blocks being loaded into some kind of file system cache the second and third times? How about if you rebooted and then ran the mimalloc version?

jq has point-free programming. it's intuitive once you wrap your head around it.

See this: https://en.wikipedia.org/wiki/Tacit_programming#jq

Re: Make Ubuntu packages 90% faster by rebuilding them

#93

Post by jeffbee »

rlpb wrote: post #38

Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…

This is generally true but specifically false. The builds described in the gist are still linking onigurama dynamically. It is in another package, libonig5, that would be updated normally.

Re: Make Ubuntu packages 90% faster by rebuilding them

#95

Post by saghm »

electromech wrote: post #79

I'd be curious how the performance compares to this Rust jq clone: cargo install --locked jaq (you might also be able to add RUSTFLAGS="-C target-cpu=native" to enable optimizations for your specific CPU family) "cargo install" is an underrated feature of Rust for exactly the kind of use case described in the article. Because it builds the tools from source, you can opt into platform-specific features/instructions th…

As a bonus that people might not be aware of, in the cases where you do want to use the repo directly (either because there isn't a published package or maybe you want the latest commit that hasn't been released), `cargo install` also has a `--git` flag that lets you specify a URL to a repo. I've used this a number of times in the past, especially as an easy way for me to quickly install personal stuff that I throw together and push to a repo without needing to put together any sort of release process or manually copy around binaries to personal machines and keep track of the exact commits I've used to build them.

Re: Make Ubuntu packages 90% faster by rebuilding them

#96

Post by Daneel_ »

tom_ wrote: post #83
ltbarcly3 wrote: post #27

Misleading title, it's 90% of the faster time. It's about 45% faster. It's actually a little bit interesting, if you are interested in how we use language. You could argue that now you now get 90% more work done in the same amount of time, and that would align with other 'speed' units that we commonly use (miles per hour, words per minute, bits per second). However, the convention in computer performance is to measur…

Related: https://randomascii.wordpress.com/2018/02/04/what-we-talk-ab...

Great read. I hadn't even considered that people might interpret "90% faster" as "10 times as fast", i.e., it will take 100-90=10% of the original time. It seems like a completely incorrect interpretation to me, but obviously there are people who read it with this understanding. Huh.

Re: Make Ubuntu packages 90% faster by rebuilding them

#97

Post by pdpi »

ryao wrote: post #76
frontfor wrote: post #54

Earlier quoted context omitted.

Are you arguing that ASLR is “security via obscurity”?

ASLR is technically a form of security by obscurity. The obscurity here being the memory layout. The reason nobody treated it that way was the high entropy that ASLR had on 64-bit, but the ASLR⊕Cache attack has undermined that significantly. You really do not want ASLR to be what determines whether an attacker takes control of your machine if you care about having a secure system.

The defining characteristic of security through obscurity is that the effectiveness of the security measure depends on the attacker not knowing about the measure at all. That description doesn’t apply to ASLR.

Re: Make Ubuntu packages 90% faster by rebuilding them

#98

Post by jmward01 »

rlpb wrote: post #38

Note that if you do this then you will opt out of any security updates not just for jq but also for its regular expression parsing dependency onigurama. For example, there was a security update for onigurama previously; if this sort of thing happens again, you'd be vulnerable, and jq is often used to parse untrusted JSON. > * SECURITY UPDATE: Fix multiple invalid pointer dereference, out-of-bounds write memory corrup…

But isn't there still the kernel of an idea here for a package management system that intelligently decides to build based on platform? Seems like a lot of performance to leave on the table.

Re: Make Ubuntu packages 90% faster by rebuilding them

#99

Post by jeffbee »

zX41ZdbW wrote: post #71

ClickHouse will be faster for processing large JSON data files. The example from the gist will look like follows: ch -q "WITH arrayJoin(features) AS f SELECT f.properties.SitusCity WHERE f.properties.TotalNetValue Reference: https://jsonbench.com/

I'm willing to believe that will execute in less than 2 seconds, but it doesn't work as given.

Re: Make Ubuntu packages 90% faster by rebuilding them

#100

Post by usefulcat »

msm_ wrote: post #69
topspin wrote: post #59

Earlier quoted context omitted.

I would, and there is no shame in it, as far as I'm concerned. I don't need to outrun the bear. I just need to outrun you.

ASLR is not security through obscurity though. It forces attacker to get a pointer leak before doing almost anything (even arbitrary read and arbitrary write primitives are useless without a leak with ASLR). As someone with a bit of experience in exploit dev, it makes a world of a difference and is one of the most influential hardenings, next to maybe stack cookies and W^X.

I'm genuinely curious what was so undesirable about this sibling comment that it was removed:

"ASLR obscures the memory layout. That is security by obscurity by definition. People thought this was okay if the entropy was high enough, but then the ASLR⊕Cache attack was published and now its usefulness is questionable."

Usually when a comment is removed, it's pretty obvious why, but in this case I'm really not seeing it at all. I read up (briefly) on the mentioned attack and can confirm that the claims made in the above comment are at the very least plausible sounding. I checked other comments from that user and don't see any other recent ones that were removed, so it doesn't seem to be a user-specific thing.

I realize this is completely off-topic, but I'd really like to understand why it was removed. Perhaps it was removed by mistake?

Post reply on HN