Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lolbench: automagically and empirically discovering Rust performance regressions

blog.anp.lol

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#31

Post by anp »

LukeShu wrote: post #24
anp wrote: post #16

Earlier quoted context omitted.

I use this word the way we did when I worked as a PC technician and help desker, where there's a lot of automation but then we sneak a bit of manual labor in to make it actually useful. Like how user accounts would be maintained in the correct state automagically.

automagically: /aw·toh·maj´i·klee/, adv. Automatically, but in a way that, for some reason (typically because it is too complicated, or too ugly, or perhaps even too trivial), the speaker doesn't feel like explaining to you. http://www.catb.org/jargon/html/A/automagically.html

Well this interpretation certainly explains some of the reactions that word has gotten in my use of it!

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#32

Post by gus_massa »

mkl wrote: post #23
MikeHolman wrote: post #22

Do you do have any plans to better distinguish between noise and regressions? I run a similar performance testing infrastructure for Chakra, and found that comparing against the previous run makes the results noisy. That means more manual review of results, which gets old fast. What I do now is run a script that averages results from the preceding 10 runs and compares that to the average of the following 5 runs to se…

Do you mean 10 preceding versions, or 10 repeated timings of the same version? If you repeat the timing for the each version many times, why is that not enough to smooth out the noise?

You usually have to kill the outliers. A cheap trick is to measure 12 times the same thing and dropping the biggest and the smallest one.

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#34

Post by habitue »

anp wrote: post #16
habitue wrote: post #13

This project looks awesome, but as a complete aside: How long do we expect it to take before "automagically" completely replaces "automatically" in English? I am guessing less than a decade to go now

I use this word the way we did when I worked as a PC technician and help desker, where there's a lot of automation but then we sneak a bit of manual labor in to make it actually useful. Like how user accounts would be maintained in the correct state automagically.

Sorry, I didn't mean to imply disapproval, it was basically an idle thought since I see that replacement pretty commonly

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#35

Post by [deleted] »

habitue wrote: post #34
anp wrote: post #16

Earlier quoted context omitted.

I use this word the way we did when I worked as a PC technician and help desker, where there's a lot of automation but then we sneak a bit of manual labor in to make it actually useful. Like how user accounts would be maintained in the correct state automagically.

Sorry, I didn't mean to imply disapproval, it was basically an idle thought since I see that replacement pretty commonly

[deleted]

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#36

Post by twtw »

panic wrote: post #33

The "More Like Rocket Science Rule of Software Engineering" has been WebKit policy for a while: https://web.archive.org/web/20061011203328/http://webkit.org... (now at https://webkit.org/performance/ ).

> Common excuses people give when they regress performance are, “But the new way is cleaner!” or “The new way is more correct.” We don’t care. No performance regressions are allowed, regardless of the reason. There is no justification for regressing performance. None.

This seems a bit extreme. Would they accept a regression to fix a critical security vulnerability? Code can be infinitely fast if it need not be correct.

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#37

Post by SatvikBeri »

twtw wrote: post #36
panic wrote: post #33

The "More Like Rocket Science Rule of Software Engineering" has been WebKit policy for a while: https://web.archive.org/web/20061011203328/http://webkit.org... (now at https://webkit.org/performance/ ).

> Common excuses people give when they regress performance are, “But the new way is cleaner!” or “The new way is more correct.” We don’t care. No performance regressions are allowed, regardless of the reason. There is no justification for regressing performance. None. This seems a bit extreme. Would they accept a regression to fix a critical security vulnerability? Code can be infinitely fast if it need not be correc…

The bigger and less centralized a team is, the less you can afford edge cases in your rules. Even running a team of 6 people I found I had to make rules more rigid than I would ideally have liked.

On my own, I'll have many internal "rules" that are very flexible. But in a team I need a small number of rules that are rigidly enforced.

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#38

Post by bloomer »

twtw wrote: post #36
panic wrote: post #33

The "More Like Rocket Science Rule of Software Engineering" has been WebKit policy for a while: https://web.archive.org/web/20061011203328/http://webkit.org... (now at https://webkit.org/performance/ ).

> Common excuses people give when they regress performance are, “But the new way is cleaner!” or “The new way is more correct.” We don’t care. No performance regressions are allowed, regardless of the reason. There is no justification for regressing performance. None. This seems a bit extreme. Would they accept a regression to fix a critical security vulnerability? Code can be infinitely fast if it need not be correc…

A better version...

Common excuses people give when introducing security vulnerabilities are, “but the new way is faster” or “the new way is more clever”. We don’t care. No security vulnerabilities are allowed, regardless of the reason. There is no justification for security vulnerabilities. None.

I love that “correct” is a “justification” in the original. I would be embarrassed to be associated with such a juvenile page. Move fast with broken things...

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#39

Post by samontar »

twtw wrote: post #36
panic wrote: post #33

The "More Like Rocket Science Rule of Software Engineering" has been WebKit policy for a while: https://web.archive.org/web/20061011203328/http://webkit.org... (now at https://webkit.org/performance/ ).

> Common excuses people give when they regress performance are, “But the new way is cleaner!” or “The new way is more correct.” We don’t care. No performance regressions are allowed, regardless of the reason. There is no justification for regressing performance. None. This seems a bit extreme. Would they accept a regression to fix a critical security vulnerability? Code can be infinitely fast if it need not be correc…

Besides I know they’ve sacrificed speed at the altar of correctness many times. My own renderer actually is much faster than theirs though at the price of a mild incorrectness: it discards all input and draws nothing.

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#40

Post by claudius »

usefulcat wrote: post #29
anp wrote: post #10

Earlier quoted context omitted.

I currently do something pretty similar by using the perf subsystem in the Linux kernel to track the behavior of each benchmark function. In my early measurements I found concurrent benchmarking to introduce unacceptable noise even with this measurement tool and with cgroups/cpusets used to pin the different processes to their own cores. Instead of trying to tune the system to account for this, I chose to build tooli…

No such 'noise' is possible with callgrind, as it's basically simulating the hardware. If you're using a VM it seems like you could still get variation between different runs due to other activity on the host system.

The problem with callgrind is (http://valgrind.org/docs/manual/cg-manual.html#branch-sim):

> Cachegrind simulates branch predictors intended to be typical of mainstream desktop/server processors of around 2004.

In other words, the data produced by Callgrind may be suitable to find obvious regressions, but there still may be more regressions which are only relevant on more modern CPUs.

Post reply on HN