This is really cool, love the project and the writeup! I regularly use nightly (I work with Rocket) and I had always wondered about this. Thank you!
Lolbench: automagically and empirically discovering Rust performance regressions
11–20 of 42 posts
Re: Lolbench: automagically and empirically discovering Rust performance regressions
#12Earlier quoted context omitted.
Each benchmark result is only compared against values from running on literally the same machine, actually. I agree that good results here would be extremely difficult to produce on virtualized infra, so I rented a few cheap dedicated servers from Hetzner. I'm glad that I decided to pin results to a single machine, because even between these identically binned machines from Hetzner I saw 2-4% variance between them wh…
A suggestion: consider using callgrind to measure performance (instructions retired, cache misses, branch mispredictions, whatever) instead of wall clock time. It will be much slower per run, but since it will also be precise you shouldn't need to do multiple runs, and you should be able to run a bunch of different benchmarks concurrently without them interfering with each other or having anything else interfere with…
Please don't, because memory access pattern will be very different.
Re: Lolbench: automagically and empirically discovering Rust performance regressions
#13This project looks awesome, but as a complete aside:
How long do we expect it to take before "automagically" completely replaces "automatically" in English?
I am guessing less than a decade to go now
Re: Lolbench: automagically and empirically discovering Rust performance regressions
#14Can I suggest you consider putting https://github.com/anp/lolbench/issues/1 in to the README.md file, so people can easily see where to look for some TODO items?
Re: Lolbench: automagically and empirically discovering Rust performance regressions
#15Can I suggest you consider putting https://github.com/anp/lolbench/issues/1 in to the README.md file, so people can easily see where to look for some TODO items?
great idea!
Re: Lolbench: automagically and empirically discovering Rust performance regressions
#16This project looks awesome, but as a complete aside: How long do we expect it to take before "automagically" completely replaces "automatically" in English? I am guessing less than a decade to go now
I use this word the way we did when I worked as a PC technician and help desker, where there's a lot of automation but then we sneak a bit of manual labor in to make it actually useful. Like how user accounts would be maintained in the correct state automagically.
Re: Lolbench: automagically and empirically discovering Rust performance regressions
#17Is there any equivalent project for java.
Java has a JIT, right? Seems more difficult to get consistent results.
Re: Lolbench: automagically and empirically discovering Rust performance regressions
#18How do you determine baseline load of the test machine in order to qualify the correctness of the benchmark? Assuming the compiling, and testing is done in the cloud how do you ensure the target platform (processor) doesn't change, and that you aren't being subjected to neighbors who are stealing RAM bandwidth, or CPU cache resources from your VM and impacting the results?
Each benchmark result is only compared against values from running on literally the same machine, actually. I agree that good results here would be extremely difficult to produce on virtualized infra, so I rented a few cheap dedicated servers from Hetzner. I'm glad that I decided to pin results to a single machine, because even between these identically binned machines from Hetzner I saw 2-4% variance between them wh…
Thanks for the link. I'll give it watch :D
Re: Lolbench: automagically and empirically discovering Rust performance regressions
#19Is there any equivalent project for java.
Java has a JIT, right? Seems more difficult to get consistent results.
You can benchmark reliably by warming up the JVM and disabling garbage collection.
https://www.ibm.com/developerworks/java/library/j-benchmark1...
https://www.ibm.com/developerworks/library/j-benchmark2/inde...
http://www.ellipticgroup.com/html/benchmarkingArticle.html
Java now ships with microbenchmarking helpers:
Re: Lolbench: automagically and empirically discovering Rust performance regressions
#20Earlier quoted context omitted.
Each benchmark result is only compared against values from running on literally the same machine, actually. I agree that good results here would be extremely difficult to produce on virtualized infra, so I rented a few cheap dedicated servers from Hetzner. I'm glad that I decided to pin results to a single machine, because even between these identically binned machines from Hetzner I saw 2-4% variance between them wh…
A suggestion: consider using callgrind to measure performance (instructions retired, cache misses, branch mispredictions, whatever) instead of wall clock time. It will be much slower per run, but since it will also be precise you shouldn't need to do multiple runs, and you should be able to run a bunch of different benchmarks concurrently without them interfering with each other or having anything else interfere with…
I haven't used callgrind, but wouldn't running benchmarks concurrently still lead to cache interference?