Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperfine: A command-line benchmarking tool

github.com

Post reply on HN
Print view View original

41–50 of 56 posts

Re: Hyperfine: A command-line benchmarking tool

#41

Post by Sesse__ »

sharkdp wrote: post #39
Sesse__ wrote: post #31

Earlier quoted context omitted.

> Hyperfine is great. Is it, though? What I would expect a system like this to have, at a minimum: * Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions) * Multiple stopping points depending on said statistics. * Automatic isolation to the greatest extent possible (given appropriate permissions) * Interleaved execution, in case something external changes mi…

> Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions) This is not included in the core of hyperfine, but we do have scripts to compute "advanced" statistics, and to perform t-tests here: https://github.com/sharkdp/hyperfine/tree/master/scripts Please feel free to comment here if you think it should be included in hyperfine itself: https://github.com/sharkd…

> This is not included in the core of hyperfine, but we do have scripts to compute "advanced" statistics, and to perform t-tests here: https://github.com/sharkdp/hyperfine/tree/master/scripts

t-tests run afoul of the “no Gaussian assumptions”, though. Distributions arising from benchmarking frequently has various forms of skew which messes up t-tests and gives artificially narrow confidence intervals.

(I'll gladly give you credit for your outlier detection, though!)

>> Automatic isolation to the greatest extent possible (given appropriate permissions) > This sounds interesting. Please feel free to open a ticket if you have any ideas.

Off the top of my head, some option that would:

* Bind to isolated CPUs, if booted with it (isolcpus=) * Binding to a consistent set of cores/hyperthreads (the scheduler frequently sabotages benchmarking, especially if your cores are have very different maximum frequency) * Warns if thermal throttling is detected during the run * Warns if an inappropriate CPU governor is enabled * Locks the program into RAM (probably hard to do without some sort of help from the program) * Enables realtime priority if available (e.g., if isolcpus= is not enabled, or you're not on Linux)

Of course, sometimes you would _want_ to benchmark some of these effects, and that's fine. But most people probably won't, and won't know that they exist. I may easily have forgotten some.

On the flip side (making things more random as opposed to less), something that randomizes the initial stack pointer would be nice, as I've sometimes seen this go really, really wrong (renaming a binary from foo to foo_new made it run >1% slower!).

Re: Hyperfine: A command-line benchmarking tool

#42

Post by renewiltord »

Sesse__ wrote: post #31
mosselman wrote: post #15

Hyperfine is great. I use it sometimes for some quick web page benchmarks: https://abuisman.com/posts/developer-tools/quick-page-benchm... As mentioned here in the thread, when you want to go into the single ms optimisations it is not the best approach since there is a lot of overhead especially the way I demonstrate here, but it works very well for some sanity checks.

> Hyperfine is great. Is it, though? What I would expect a system like this to have, at a minimum: * Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions) * Multiple stopping points depending on said statistics. * Automatic isolation to the greatest extent possible (given appropriate permissions) * Interleaved execution, in case something external changes mi…

Personally, I'm all about the UNIX philosophy of doing one thing and doing it well. All I want is the process to be invoked k times to do a thing with warmup etc. etc. If I want additional stats, it's easy to calculate. I just `--export-json` and then once it's in a dataframe I can do what I want with it.

Re: Hyperfine: A command-line benchmarking tool

#43

Post by renewiltord »

mmastrac wrote: post #34
sharkdp wrote: post #16

Earlier quoted context omitted.

Caching is something that you almost always have to be aware of when benchmarking command line applications, even if the application itself has no caching behavior. Please see https://github.com/sharkdp/hyperfine?tab=readme-ov-file#warm... on how to run either warm-cache benchmarks or cold-cache benchmarks.

I'm fully aware but it's not a problem that warmup runs fix. An executable freshly compiled will always benchmark differently than one that has "cooled off" on macos, regardless of warmup runs. I've tried to understand what the issue is (played with resigning executables etc) but it's literally something about the inode of the executable itself. Most likely part of the OSX security system.

Interesting. I've encountered this obviously on first run (because of the security checking it does on novel executables) but didn't realize this expired. Probably because I usually attribute it to a recompilation. Thanks.

Re: Hyperfine: A command-line benchmarking tool

#44

Post by sharkdp »

Sesse__ wrote: post #41
sharkdp wrote: post #39

Earlier quoted context omitted.

> Robust statistics with p-values (not just min/max, compensation for multiple hypotheses, no Gaussian assumptions) This is not included in the core of hyperfine, but we do have scripts to compute "advanced" statistics, and to perform t-tests here: https://github.com/sharkdp/hyperfine/tree/master/scripts Please feel free to comment here if you think it should be included in hyperfine itself: https://github.com/sharkd…

> This is not included in the core of hyperfine, but we do have scripts to compute "advanced" statistics, and to perform t-tests here: https://github.com/sharkdp/hyperfine/tree/master/scripts t-tests run afoul of the “no Gaussian assumptions”, though. Distributions arising from benchmarking frequently has various forms of skew which messes up t-tests and gives artificially narrow confidence intervals. (I'll gladly gi…

> On the flip side (making things more random as opposed to less), something that randomizes the initial stack pointer would be nice, as I've sometimes seen this go really, really wrong (renaming a binary from foo to foo_new made it run >1% slower!).

This is something we do already. We set a `HYPERFINE_RANDOMIZED_ENVIRONMENT_OFFSET` environment variable with a random-length value: https://github.com/sharkdp/hyperfine/blob/87d77c861f1b6c761a...

Re: Hyperfine: A command-line benchmarking tool

#45

Post by fwip »

forrestthewoods wrote: post #36
sharkdp wrote: post #18

Earlier quoted context omitted.

That doesn't make a lot of sense. It's more like the opposite of what you are saying. The precision of hyperfine is typically in the single-digit millisecond range. Maybe just below 1 ms if you take special care to run the benchmark on a quiet system. Everything below that (microsecond or nanosecond range) is something that you need to address with other forms of benchmarking. But for everything in the right range (m…

No it’s not. Back in the day my goal for Advent of Code was to run all solutions in under 1 second total. Hyperfine would take like 30 minutes to benchmark a 1 second runtime. It was hyper frustrating. I could not find a good way to get Hyperfine to do what I wanted.

I've been using it for about four or five years, and never experienced this behavior.

Current defaults: "By default, it will perform at least 10 benchmarking runs and measure for at least 3 seconds." If your program takes 1s to run, it should take 10 seconds to benchmark.

Is it possible that your program was waiting for input that never came? One "gotcha" is that it expects each argument to be a full program, so if you ran `hyperfine ./a.out input.txt`, it will first bench a.out with no args, then try to bench input.txt (which will fail). If a.out reads from stdin when no argument is given, then it would hang forever, and I can see why you'd give up after a half hour.

Re: Hyperfine: A command-line benchmarking tool

#46

Post by ratrocket »

Perhaps interesting (for some) to note that hyperfine is from the same author as at least a few other "ne{w,xt} generation" command line tools (that could maybe be seen as part of "rewrite it in Rust", but I don't want to paint the author with a brush they disagree with!!): fd (find alternative; https://github.com/sharkdp/fd), bat ("supercharged version of the cat command"; https://github.com/sharkdp/bat), and hexyl (hex viewer; https://github.com/sharkdp/hexyl). (And certainly others I've missed!)

Pointing this out because I myself appreciate comments that do this.

For myself, `fd` is the one most incorporated into my own "toolbox" -- used it this morning prior to seeing this thread on hyperfine! So, thanks for all that, sharkdp if you're reading!

Ok, end OT-ness.

Re: Hyperfine: A command-line benchmarking tool

#47

Post by sharkdp »

fwip wrote: post #45
forrestthewoods wrote: post #36

Earlier quoted context omitted.

No it’s not. Back in the day my goal for Advent of Code was to run all solutions in under 1 second total. Hyperfine would take like 30 minutes to benchmark a 1 second runtime. It was hyper frustrating. I could not find a good way to get Hyperfine to do what I wanted.

I've been using it for about four or five years, and never experienced this behavior. Current defaults: "By default, it will perform at least 10 benchmarking runs and measure for at least 3 seconds." If your program takes 1s to run, it should take 10 seconds to benchmark. Is it possible that your program was waiting for input that never came? One "gotcha" is that it expects each argument to be a full program, so if y…

> Is it possible that your program was waiting for input that never came?

We do close stdin to prevent this. So you can benchmark `cat`, for example, and it works just fine.

Re: Hyperfine: A command-line benchmarking tool

#48

Post by varenc »

ratrocket wrote: post #46

Perhaps interesting (for some) to note that hyperfine is from the same author as at least a few other "ne{w,xt} generation" command line tools (that could maybe be seen as part of "rewrite it in Rust", but I don't want to paint the author with a brush they disagree with!!): fd (find alternative; https://github.com/sharkdp/fd ), bat ("supercharged version of the cat command"; https://github.com/sharkdp/bat ), and hexy…

++ to `fd`

It’s absolutely my preferred `find` replacement. Its CLI interface just clicks for me and I can quickly express my desires. Quite unlike `find`. `fd` is one of the first packages I install on a new system.

Re: Hyperfine: A command-line benchmarking tool

#49

Post by jiehong »

llimllib wrote: post #28
mosselman wrote: post #15

Hyperfine is great. I use it sometimes for some quick web page benchmarks: https://abuisman.com/posts/developer-tools/quick-page-benchm... As mentioned here in the thread, when you want to go into the single ms optimisations it is not the best approach since there is a lot of overhead especially the way I demonstrate here, but it works very well for some sanity checks.

I find k6 a lot nicer for HTTP benching, and no slower to set up than hyperfine (which I love for CLI benching): https://k6.io/

Could hyperfine running curl be an alternative?

Re: Hyperfine: A command-line benchmarking tool

#50

Post by fwip »

sharkdp wrote: post #47
fwip wrote: post #45

Earlier quoted context omitted.

I've been using it for about four or five years, and never experienced this behavior. Current defaults: "By default, it will perform at least 10 benchmarking runs and measure for at least 3 seconds." If your program takes 1s to run, it should take 10 seconds to benchmark. Is it possible that your program was waiting for input that never came? One "gotcha" is that it expects each argument to be a full program, so if y…

> Is it possible that your program was waiting for input that never came? We do close stdin to prevent this. So you can benchmark `cat`, for example, and it works just fine.

Oh, my bad! Thank you for the correction, and for all your work making hyperfine.
Post reply on HN