Show HN: Hyperfine – a command-line benchmarking tool
1–10 of 26 posts
Re: Show HN: Hyperfine – a command-line benchmarking tool
#2I have submitted "hyperfine" 1.5 years ago when it just came out. Since then, the program has gained functionality (statistical outlier detection, result export, parametrized benchmarks) and maturity.
Old discussion: https://news.ycombinator.com/item?id=16193225
Looking forward to your feedback!
Re: Show HN: Hyperfine – a command-line benchmarking tool
#3I started using hyperfine a few months ago now on a colleague’s recommendation and I really like it.
In the past, I’ve cobbled together quick bash pipelines to run time in a loop, awk out timings, and compute averages, but it was always a pain. Hyperfine has a great interface and really useful reports. It actually reminds me quite a bit of Criterion, the benchmarking suite for Rust.
I also use fd and bat extensively, so thanks for making such useful tools!
Re: Show HN: Hyperfine – a command-line benchmarking tool
#4I started using hyperfine a few months ago now on a colleague’s recommendation and I really like it. In the past, I’ve cobbled together quick bash pipelines to run time in a loop, awk out timings, and compute averages, but it was always a pain. Hyperfine has a great interface and really useful reports. It actually reminds me quite a bit of Criterion, the benchmarking suite for Rust. I also use fd and bat extensively,…
Thank you very much for your feedback!
Re: Show HN: Hyperfine – a command-line benchmarking tool
#5I have submitted "hyperfine" 1.5 years ago when it just came out. Since then, the program has gained functionality (statistical outlier detection, result export, parametrized benchmarks) and maturity. Old discussion: https://news.ycombinator.com/item?id=16193225 Looking forward to your feedback!
Since you cite bench as an inspiration, have you ever thought about including the nice graphical HTML page with graphs that bench outputs? In a similar vein what are your thoughts on directly depending on and using criterion (the Rust port)?
Re: Show HN: Hyperfine – a command-line benchmarking tool
#6Most -- nearly all -- benchmarking tools like this work from a normality assumption, i.e. assume that results follow the normal distribution, or is close to it. Some do this on blind faith, others argue from the CLT that "with infinite samples, the mean is normally distributed, so surely it must be also with finite number of samples, at least a little?"
In fact, performance numbers (latencies) often follow a heavy-tailed distribution. For these, you need a literal shitload of samples to get even a slightly normal mean. For these, the sample mean, the sample variance, the sample centiles -- they all severely underestimate the true values.
What's worse is when these tools start to remove "outliers". With a heavy-tailed distribution, the majority of samples don't contribute very much at all to the expectation. The strongest signal is found in the extreme values. The strongest signal is found in the stuff that is thrown out. The junk that's left is the noise, the stuff that doesn't tell you very much about what you're dealing with.
I stand firm in my belief that unless you can prove how CLT applies to your input distributions, you should not assume normality.
And if you don't know what you are doing, stop reporting means. Stop reporting centiles. Report the maximum value. That's a really boring thing to hear, but it is nearly always statistically and analytically meaningful, so it is a good default.
Re: Show HN: Hyperfine – a command-line benchmarking tool
#7Most -- nearly all -- benchmarking tools like this work from a normality assumption, i.e. assume that results follow the normal distribution, or is close to it. Some do this on blind faith, others argue from the CLT that "with infinite samples, the mean is normally distributed, so surely it must be also with finite number of samples, at least a little?" In fact, performance numbers (latencies) often follow a heavy-ta…
> Report the maximum value.
In benchmarks, assuming you run the same workload each time, you often want the minimum value. Anything else just tells you how much system overhead you encountered.
(Complete agreement that applying statistics without knowing anything about the distribution can mislead.)
Re: Show HN: Hyperfine – a command-line benchmarking tool
#8Most -- nearly all -- benchmarking tools like this work from a normality assumption, i.e. assume that results follow the normal distribution, or is close to it. Some do this on blind faith, others argue from the CLT that "with infinite samples, the mean is normally distributed, so surely it must be also with finite number of samples, at least a little?" In fact, performance numbers (latencies) often follow a heavy-ta…
I'll take some sparkline histograms in the console, too.
Re: Show HN: Hyperfine – a command-line benchmarking tool
#9Most -- nearly all -- benchmarking tools like this work from a normality assumption, i.e. assume that results follow the normal distribution, or is close to it. Some do this on blind faith, others argue from the CLT that "with infinite samples, the mean is normally distributed, so surely it must be also with finite number of samples, at least a little?" In fact, performance numbers (latencies) often follow a heavy-ta…
> Report the maximum value. In benchmarks, assuming you run the same workload each time, you often want the minimum value. Anything else just tells you how much system overhead you encountered. (Complete agreement that applying statistics without knowing anything about the distribution can mislead.)
To be pedantic, you want the mode of the distribution. You are right though, the minimum is a much closer approximation of the mode in most situations.
Re: Show HN: Hyperfine – a command-line benchmarking tool
#10Most -- nearly all -- benchmarking tools like this work from a normality assumption, i.e. assume that results follow the normal distribution, or is close to it. Some do this on blind faith, others argue from the CLT that "with infinite samples, the mean is normally distributed, so surely it must be also with finite number of samples, at least a little?" In fact, performance numbers (latencies) often follow a heavy-ta…
If your distribution is heavy-tailed, then in order to find the mean (or any other measure of "location") you typically need to pay _less_ attention to outliers, not _more_.
So, e.g.: for a normal distribution, with very narrow tails -- probabilities like exp(-x^2) -- your sample mean is the maximum-likelihood estimator. For a double-exponential (Laplace) distribution, whose tails are like exp(-|x|) and therefore much fatter, the maximum-likelihood estimator is the _median_, which gives much less weight to outliers. Another way to look at it: the mean minimizes the sum of |x-m|^2 and the median minimizes the sum of |x-m|, and the former grows in size and hence importance much faster as x gets large.