Show HN: Hyperfine – a command-line benchmarking tool

Thank you for the feedback. I agree with most of your points.

> Most -- nearly all -- benchmarking tools like this work from a normality assumption

I don't think that hyperfine makes any assumption about normality. Sure, we do report sample mean and sample standard deviation by default, but we also report sample minimum and the maximum. You can also easily export all the benchmark results and inspect in more detail with the supplied Python scripts.

> In fact, performance numbers (latencies) often follow a heavy-tailed distribution

So when is this really the case? In my understanding, if I am measuring the runtime of a deterministic program with the same input, the runtime should only be influenced by external factors that are out of my control (other programs being scheduled, caching effects, hardware-specific influences, ..). These are exactly the things that I want to "average out" by running the benchmark multiple times.

> What's worse is when these tools start to remove "outliers".

Hyperfine never removes outliers. What we do is to try and detect outliers. We do this by computing robust statistical estimates that specifically DO NOT assume a normal distribution (see https://github.com/sharkdp/hyperfine/blob/master/src/hyperfi... for details).

We perform this outlier detection to warn users about potentially interfering processes or caching effects.

Take a look at these results, for example: https://i.imgur.com/XRvE6Ys.png

I benchmarked a file-searching program. The underlying distribution, while probably not normal, seems to be "well behaved" and I think that the sample mean and the sample standard deviation could be quantities with a reasonably predictive power.

What you do NOT see in the histogram is a single outlier at 1.15 seconds, far outside the plot to the right. This was the first benchmark run where the disk caches were still cold. In such a case, hyperfine warns the user:

Warning: The first benchmarking run for this command was significantly slower than the rest (1.152 s). This could be caused by (filesystem) caches that were not filled until after the first run. You should consider using the '--warmup' option to fill those caches before the actual benchmark. Alternatively, use the '--prepare' option to clear the caches before each timing run.

In conclusion, I am not quite sure how your critisism applies to hyperfine, but I'd be happy to get further feedback.

Ah those were earnest questions, not feature requests in disguise :), especially the second question. I haven't so far had any need for either of those things while using hyperfine (although admittedly I've only used hyperfine for about 3 small projects).

I ask purely as a comparison to some of the design choices bench made for personal edification.

That being said, one of the hypothetical advantages of using criterion is that you get to piggy-back on its visualizations and statistical analyses, which are quite useful for showing friends and coworkers. I'm not sure of the specifics of how you're doing outlier detection, but Criterion's method is quite nice and I find the choice of linear regression for linearly increasing iterations to be an interesting sanity check.

Criterion can work for lower iterations as well (although tens is definitely getting into the lower bound).

I agree that the maximum is useful, but in a situation where you're optimizing code on a machine where there is external interference from other processes, the minimum can be more relevant.

This is when the code you write is deterministic and the interference is not. The minimum is closer to what you would get without interference.

Just don't effect the time to represent typical results. (And why would you expect that if you're running benchmarks on your development machine?)

Just looking at the chart, your article makes a case that minimum is much better than maximum, and in fact if you report one number, the minimum is the best number to report (in that particular example).

If we go into details - sure, 1 number is not ideal, but neither is the confidence interval, because people will assume normal distribution for them. If you report 2 numbers, perhaps instead of confidence interval (which falsely implies normal distribution) it's better to report the mode and the median.

Very cool! I'd love to reference this in the hyperfine README