Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperfine: A command-line benchmarking tool

github.com

Post reply on HN
Print view View original

31–40 of 56 posts

Re: Hyperfine: A command-line benchmarking tool

#31

Post by Sesse__ »

mosselman wrote: post #15

Hyperfine is great. I use it sometimes for some quick web page benchmarks: https://abuisman.com/posts/developer-tools/quick-page-benchm... As mentioned here in the thread, when you want to go into the single ms optimisations it is not the best approach since there is a lot of overhead especially the way I demonstrate here, but it works very well for some sanity checks.

> Hyperfine is great.

Is it, though?

What I would expect a system like this to have, at a minimum:

  * Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions)
  * Multiple stopping points depending on said statistics.
  * Automatic isolation to the greatest extent possible (given appropriate permissions)
  * Interleaved execution, in case something external changes mid-way.
I don't see any of this in hyperfine. It just… runs things N times and then does a naïve average/min/max? At that rate, one could just as well use a shell script and eyeball the results.

Re: Hyperfine: A command-line benchmarking tool

#32

Post by bee_rider »

Sesse__ wrote: post #31
mosselman wrote: post #15

Hyperfine is great. I use it sometimes for some quick web page benchmarks: https://abuisman.com/posts/developer-tools/quick-page-benchm... As mentioned here in the thread, when you want to go into the single ms optimisations it is not the best approach since there is a lot of overhead especially the way I demonstrate here, but it works very well for some sanity checks.

> Hyperfine is great. Is it, though? What I would expect a system like this to have, at a minimum: * Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions) * Multiple stopping points depending on said statistics. * Automatic isolation to the greatest extent possible (given appropriate permissions) * Interleaved execution, in case something external changes mi…

What do you suggest? Those sound like great features.

Re: Hyperfine: A command-line benchmarking tool

#33

Post by Sesse__ »

bee_rider wrote: post #32
Sesse__ wrote: post #31

Earlier quoted context omitted.

> Hyperfine is great. Is it, though? What I would expect a system like this to have, at a minimum: * Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions) * Multiple stopping points depending on said statistics. * Automatic isolation to the greatest extent possible (given appropriate permissions) * Interleaved execution, in case something external changes mi…

What do you suggest? Those sound like great features.

I've only seen such things in internal tools so far, unfortunately, so if you see anything in public, please tell me :-) I'm just confused why everything thinks hyperfine is so awesome, when it does not meet what I'd consider a fairly low bar for benchmarking tools? (“Best publicly available” != “great”, in my book.)

Re: Hyperfine: A command-line benchmarking tool

#34

Post by mmastrac »

sharkdp wrote: post #16
mmastrac wrote: post #2

Hyperfine is a great tool but when I was using it at Deno to benchmark startup time there was a lot of weirdness around the operating system apparently caching inodes of executables. If you are looking at shaving sub 20ms numbers, be aware you may need to pull tricks on macos especially to get real numbers.

Caching is something that you almost always have to be aware of when benchmarking command line applications, even if the application itself has no caching behavior. Please see https://github.com/sharkdp/hyperfine?tab=readme-ov-file#warm... on how to run either warm-cache benchmarks or cold-cache benchmarks.

I'm fully aware but it's not a problem that warmup runs fix. An executable freshly compiled will always benchmark differently than one that has "cooled off" on macos, regardless of warmup runs.

I've tried to understand what the issue is (played with resigning executables etc) but it's literally something about the inode of the executable itself. Most likely part of the OSX security system.

Re: Hyperfine: A command-line benchmarking tool

#36

Post by forrestthewoods »

sharkdp wrote: post #18
forrestthewoods wrote: post #5

Hyperfine is hyper frustrating because it only works with really really fine microsecond level benchmarks. Once you get into the millisecond range it’s worthless.

That doesn't make a lot of sense. It's more like the opposite of what you are saying. The precision of hyperfine is typically in the single-digit millisecond range. Maybe just below 1 ms if you take special care to run the benchmark on a quiet system. Everything below that (microsecond or nanosecond range) is something that you need to address with other forms of benchmarking. But for everything in the right range (m…

No it’s not.

Back in the day my goal for Advent of Code was to run all solutions in under 1 second total. Hyperfine would take like 30 minutes to benchmark a 1 second runtime.

It was hyper frustrating. I could not find a good way to get Hyperfine to do what I wanted.

Re: Hyperfine: A command-line benchmarking tool

#37

Post by sharkdp »

forrestthewoods wrote: post #36
sharkdp wrote: post #18

Earlier quoted context omitted.

That doesn't make a lot of sense. It's more like the opposite of what you are saying. The precision of hyperfine is typically in the single-digit millisecond range. Maybe just below 1 ms if you take special care to run the benchmark on a quiet system. Everything below that (microsecond or nanosecond range) is something that you need to address with other forms of benchmarking. But for everything in the right range (m…

No it’s not. Back in the day my goal for Advent of Code was to run all solutions in under 1 second total. Hyperfine would take like 30 minutes to benchmark a 1 second runtime. It was hyper frustrating. I could not find a good way to get Hyperfine to do what I wanted.

If that's the case, I would consider it a bug. Please feel free to report it. In general, hyperfine should not take longer than ~3 seconds, unless the command itself takes > 300 ms second to run. In the latter case, we do a minimum of 10 runs by default. So if your program takes 3 min for a single iteration, it would take 30 min by default — yes. But this can be controlled using the `-m`/`--min-runs` option. You can also specify the exact amount of runs using `-r`/`--runs`, if you prefer that.

> I could not find a good way to get Hyperfine to do what I wanted

This is all documented here: https://github.com/sharkdp/hyperfine/tree/master?tab=readme-... under "Basic benchmarks". The options to control the amount of runs are also listed in `hyperfine --help` and in the man page. Please let us know if you think we can improve the documentation / discovery of those options.

Re: Hyperfine: A command-line benchmarking tool

#39

Post by sharkdp »

Sesse__ wrote: post #31
mosselman wrote: post #15

Hyperfine is great. I use it sometimes for some quick web page benchmarks: https://abuisman.com/posts/developer-tools/quick-page-benchm... As mentioned here in the thread, when you want to go into the single ms optimisations it is not the best approach since there is a lot of overhead especially the way I demonstrate here, but it works very well for some sanity checks.

> Hyperfine is great. Is it, though? What I would expect a system like this to have, at a minimum: * Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions) * Multiple stopping points depending on said statistics. * Automatic isolation to the greatest extent possible (given appropriate permissions) * Interleaved execution, in case something external changes mi…

> Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions)

This is not included in the core of hyperfine, but we do have scripts to compute "advanced" statistics, and to perform t-tests here: https://github.com/sharkdp/hyperfine/tree/master/scripts

Please feel free to comment here if you think it should be included in hyperfine itself: https://github.com/sharkdp/hyperfine/issues/523

> Automatic isolation to the greatest extent possible (given appropriate permissions)

This sounds interesting. Please feel free to open a ticket if you have any ideas.

> Interleaved execution, in case something external changes mid-way.

Please see the discussion here: https://github.com/sharkdp/hyperfine/issues/21

> It just… runs things N times and then does a naïve average/min/max?

While there is nothing wrong with computing average/min/max, this is not all hyperfine does. We also compute modified Z-scores to detect outliers. We use that to issue warnings, if we think the mean value is influenced by them. We also warn if the first run of a command took significantly longer than the rest of the runs and suggest counter-measures.

Depending on the benchmark I do, I tend to look at either the `min` or the `mean`. If I need something more fine-grained, I export the results and use the scripts referenced above.

> At that rate, one could just as well use a shell script and eyeball the results.

Statistical analysis (which you can consider to be basic) is just one reason why I wrote hyperfine. The other reason is that I wanted to make benchmarking easy to use. I use warmup runs, preparation commands and parametrized benchmarks all the time. I also frequently use the Markdown export or the JSON export to generate graphs or histograms. This is my personal experience. If you are not interested in all of these features, you can obviously "just as well use a shell script".

Re: Hyperfine: A command-line benchmarking tool

#40

Post by sharkdp »

Sesse__ wrote: post #33
bee_rider wrote: post #32

Earlier quoted context omitted.

What do you suggest? Those sound like great features.

I've only seen such things in internal tools so far, unfortunately, so if you see anything in public, please tell me :-) I'm just confused why everything thinks hyperfine is so awesome, when it does not meet what I'd consider a fairly low bar for benchmarking tools? (“Best publicly available” != “great”, in my book.)

> “Best publicly available” != “great”

Of course. But it is free and open source. And everyone is invited to make it better.

Post reply on HN