Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Lolbench: automagically and empirically discovering Rust performance regressions

blog.anp.lol

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#21

Post by shepmaster »

usefulcat wrote: post #9
anp wrote: post #8

Earlier quoted context omitted.

Each benchmark result is only compared against values from running on literally the same machine, actually. I agree that good results here would be extremely difficult to produce on virtualized infra, so I rented a few cheap dedicated servers from Hetzner. I'm glad that I decided to pin results to a single machine, because even between these identically binned machines from Hetzner I saw 2-4% variance between them wh…

A suggestion: consider using callgrind to measure performance (instructions retired, cache misses, branch mispredictions, whatever) instead of wall clock time. It will be much slower per run, but since it will also be precise you shouldn't need to do multiple runs, and you should be able to run a bunch of different benchmarks concurrently without them interfering with each other or having anything else interfere with…

This is one of the many metrics of the official Rust compiler performance benchmarks [1].

[1]: https://perf.rust-lang.org/nll-dashboard.html

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#22

Post by MikeHolman »

Do you do have any plans to better distinguish between noise and regressions? I run a similar performance testing infrastructure for Chakra, and found that comparing against the previous run makes the results noisy. That means more manual review of results, which gets old fast.

What I do now is run a script that averages results from the preceding 10 runs and compares that to the average of the following 5 runs to see if the regression is consistent or anomalous. If the regression is consistent, then the script automatically files a bug in our tracker.

There is still some noise in the results, but it cuts down on those one-off issues.

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#23

Post by mkl »

MikeHolman wrote: post #22

Do you do have any plans to better distinguish between noise and regressions? I run a similar performance testing infrastructure for Chakra, and found that comparing against the previous run makes the results noisy. That means more manual review of results, which gets old fast. What I do now is run a script that averages results from the preceding 10 runs and compares that to the average of the following 5 runs to se…

Do you mean 10 preceding versions, or 10 repeated timings of the same version? If you repeat the timing for the each version many times, why is that not enough to smooth out the noise?

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#24

Post by LukeShu »

anp wrote: post #16
habitue wrote: post #13

This project looks awesome, but as a complete aside: How long do we expect it to take before "automagically" completely replaces "automatically" in English? I am guessing less than a decade to go now

I use this word the way we did when I worked as a PC technician and help desker, where there's a lot of automation but then we sneak a bit of manual labor in to make it actually useful. Like how user accounts would be maintained in the correct state automagically.

automagically: /aw·toh·maj´i·klee/, adv. Automatically, but in a way that, for some reason (typically because it is too complicated, or too ugly, or perhaps even too trivial), the speaker doesn't feel like explaining to you.

http://www.catb.org/jargon/html/A/automagically.html

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#25

Post by LukeShu »

anothergoogler wrote: post #19
dralley wrote: post #17

Earlier quoted context omitted.

Java has a JIT, right? Seems more difficult to get consistent results.

You can benchmark reliably by warming up the JVM and disabling garbage collection. https://www.ibm.com/developerworks/java/library/j-benchmark1... https://www.ibm.com/developerworks/library/j-benchmark2/inde... http://www.ellipticgroup.com/html/benchmarkingArticle.html Java now ships with microbenchmarking helpers: http://openjdk.java.net/projects/code-tools/jmh/

https://tratt.net/laurie/blog/entries/why_arent_more_users_m...

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#26

Post by CUViper »

usefulcat wrote: post #9
anp wrote: post #8

Earlier quoted context omitted.

Each benchmark result is only compared against values from running on literally the same machine, actually. I agree that good results here would be extremely difficult to produce on virtualized infra, so I rented a few cheap dedicated servers from Hetzner. I'm glad that I decided to pin results to a single machine, because even between these identically binned machines from Hetzner I saw 2-4% variance between them wh…

A suggestion: consider using callgrind to measure performance (instructions retired, cache misses, branch mispredictions, whatever) instead of wall clock time. It will be much slower per run, but since it will also be precise you shouldn't need to do multiple runs, and you should be able to run a bunch of different benchmarks concurrently without them interfering with each other or having anything else interfere with…

Some of those target benchmarks are on Rayon, and we've found that valgrind interferes with threading way too much to be useful there.

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#27

Post by usefulcat »

MikeHolman wrote: post #20
usefulcat wrote: post #9

Earlier quoted context omitted.

A suggestion: consider using callgrind to measure performance (instructions retired, cache misses, branch mispredictions, whatever) instead of wall clock time. It will be much slower per run, but since it will also be precise you shouldn't need to do multiple runs, and you should be able to run a bunch of different benchmarks concurrently without them interfering with each other or having anything else interfere with…

I haven't used callgrind, but wouldn't running benchmarks concurrently still lead to cache interference?

No, because callgrind is simulating the hardware, including the caches. Which is why it's also much slower.

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#28

Post by daurnimator »

anp wrote: post #16
habitue wrote: post #13

This project looks awesome, but as a complete aside: How long do we expect it to take before "automagically" completely replaces "automatically" in English? I am guessing less than a decade to go now

I use this word the way we did when I worked as a PC technician and help desker, where there's a lot of automation but then we sneak a bit of manual labor in to make it actually useful. Like how user accounts would be maintained in the correct state automagically.

I've used it for decades to mean "automatically as if by magic" in the sense of Arthur C Clake's quote: "Any sufficiently advanced technology is indistinguishable from magic."

To flip this adage around: calling your own tech as performing something "automagically" is tantamount to calling it "sufficiently advanced technology".

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#29

Post by usefulcat »

anp wrote: post #10
usefulcat wrote: post #9

Earlier quoted context omitted.

A suggestion: consider using callgrind to measure performance (instructions retired, cache misses, branch mispredictions, whatever) instead of wall clock time. It will be much slower per run, but since it will also be precise you shouldn't need to do multiple runs, and you should be able to run a bunch of different benchmarks concurrently without them interfering with each other or having anything else interfere with…

I currently do something pretty similar by using the perf subsystem in the Linux kernel to track the behavior of each benchmark function. In my early measurements I found concurrent benchmarking to introduce unacceptable noise even with this measurement tool and with cgroups/cpusets used to pin the different processes to their own cores. Instead of trying to tune the system to account for this, I chose to build tooli…

No such 'noise' is possible with callgrind, as it's basically simulating the hardware. If you're using a VM it seems like you could still get variation between different runs due to other activity on the host system.

Re: Lolbench: automagically and empirically discovering Rust performance regressions

#30

Post by anp »

MikeHolman wrote: post #22

Do you do have any plans to better distinguish between noise and regressions? I run a similar performance testing infrastructure for Chakra, and found that comparing against the previous run makes the results noisy. That means more manual review of results, which gets old fast. What I do now is run a script that averages results from the preceding 10 runs and compares that to the average of the following 5 runs to se…

I talked about this a little bit in the meetup talk I linked and I intend to write a bit more about this, but I'll try to summarize.

There are kind of three prongs here:

First, using criterion.rs does a ton for giving us more stable metrics. It handles things like warmups, accounting for obvious statistical outliers in the sample runs, postprocessing the raw data to provide more meaningful statistics, etc. I'm currently using a fork of the library which additionally does this recording and processing of a variety of metrics we get from `perf_event_open` on Linux but which I assume you could get through ETW or Intel/AMD's userspace PMC libraries.

Second, I try to provide a stable environment so that results over long time deltas are comparable and we can store the data for offline analysis rather than having to checkout recent prior commits and compare the current PR/nightly/etc against them. Prior to the current deployment I was using cgroups to move potentially competing processes off of the benchmark cores which produced some nice results. However I had some issues with the version of the cpuset utility I installed on the debian machines and I haven't sorted it out yet.

Third, we do a few things with the time-series-esque data we get from measuring multiple toolchains to try and only surface relevant results. Those are mostly in src/analysis.rs if you want to poke around. It basically boils down to calculating the Kernel Density Estimate of the current toolchain's value being from the same population (I hope these terms are halfway correct) as all prior toolchains' value.

I hope that with a few extensions to the above we can get close to being reliable enough to include in early PR feedback, but I think the likely best case scenario is a manually invoked bot on PRs followed by me and a few other people triaging the regressions surfaced by the tool after something merges.

Here are a few issues that I think will help improve this too:

https://github.com/anp/lolbench/issues/20

https://github.com/anp/lolbench/issues/17

https://github.com/anp/lolbench/issues/14

Post reply on HN