Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Hyperfine: A command-line benchmarking tool

github.com

Post reply on HN
Print view View original

11–20 of 56 posts

Re: Hyperfine: A command-line benchmarking tool

#11

Post by PhilipRoman »

forrestthewoods wrote: post #7
anotherhue wrote: post #6

Earlier quoted context omitted.

It spawns a new process each time right? I would think that would but a cap on how accurate it can get. For my purposes I use it all the time though, quick and easy sanity-check.

The issue is it runs a kajillion tests to try and be “statistical”. But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”. It’s very much designed for nanosecond to low microsecond benchmarks. Trying to fight this is trying to smash a square peg through a round hole.

I disagree that it is designed for nano/micro benchmarks. If you want that level of detail, you need to stay within a single process, pinned to a core which is isolated from scheduler. At least I found it almost impossible to benchmark assembly routines with it.

Re: Hyperfine: A command-line benchmarking tool

#12

Post by art049 »

JackYoustra wrote: post #4
mmastrac wrote: post #2

Hyperfine is a great tool but when I was using it at Deno to benchmark startup time there was a lot of weirdness around the operating system apparently caching inodes of executables. If you are looking at shaving sub 20ms numbers, be aware you may need to pull tricks on macos especially to get real numbers.

I've found pretty good results with the System Trace template in xcode instruments. You can also stack instruments, for example combining the file inspector with a virtual memory inspector. I've run into some memory corruption with it sometimes, though, so be wary of that. Emerge tools has an alternative for iOS at least, maybe one day they'll port it to mac.

I never tried xcode instruments. Is the UX good for this kind of tool?

Re: Hyperfine: A command-line benchmarking tool

#13

Post by usrme »

I've also had a good experience using the 'perf'[^1] tools for when I don't want to install 'hyperfine'. Shameless plug for a small blog post about it as I don't think it is that well known: https://usrme.xyz/tils/perf-is-more-robust-for-repeated-timi....

---

[^1]: https://www.mankier.com/1/perf

Re: Hyperfine: A command-line benchmarking tool

#14

Post by vdm »

usrme wrote: post #13

I've also had a good experience using the 'perf'[^1] tools for when I don't want to install 'hyperfine'. Shameless plug for a small blog post about it as I don't think it is that well known: https://usrme.xyz/tils/perf-is-more-robust-for-repeated-timi... . --- [^1]: https://www.mankier.com/1/perf

I too have scripted time(1) in a loop badly. perf stat is more likely to be already installed than hyperfine. Thank you for sharing!

Re: Hyperfine: A command-line benchmarking tool

#15

Post by mosselman »

Hyperfine is great. I use it sometimes for some quick web page benchmarks:

https://abuisman.com/posts/developer-tools/quick-page-benchm...

As mentioned here in the thread, when you want to go into the single ms optimisations it is not the best approach since there is a lot of overhead especially the way I demonstrate here, but it works very well for some sanity checks.

Re: Hyperfine: A command-line benchmarking tool

#16

Post by sharkdp »

mmastrac wrote: post #2

Hyperfine is a great tool but when I was using it at Deno to benchmark startup time there was a lot of weirdness around the operating system apparently caching inodes of executables. If you are looking at shaving sub 20ms numbers, be aware you may need to pull tricks on macos especially to get real numbers.

Caching is something that you almost always have to be aware of when benchmarking command line applications, even if the application itself has no caching behavior. Please see https://github.com/sharkdp/hyperfine?tab=readme-ov-file#warm... on how to run either warm-cache benchmarks or cold-cache benchmarks.

Re: Hyperfine: A command-line benchmarking tool

#17

Post by sharkdp »

PhilipRoman wrote: post #10
oguz-ismail wrote: post #8

Earlier quoted context omitted.

It spawns a new shell for each run and subtracts the average shell startup time from final results. Too much noise

The shell can be disabled, leaving just fork+exec

Yes. If you don't make use of shell builtins/syntax, you can use hyperfine's `--shell=none`/`-N` option to disable the intermediate shell.

Re: Hyperfine: A command-line benchmarking tool

#18

Post by sharkdp »

forrestthewoods wrote: post #5

Hyperfine is hyper frustrating because it only works with really really fine microsecond level benchmarks. Once you get into the millisecond range it’s worthless.

That doesn't make a lot of sense. It's more like the opposite of what you are saying. The precision of hyperfine is typically in the single-digit millisecond range. Maybe just below 1 ms if you take special care to run the benchmark on a quiet system. Everything below that (microsecond or nanosecond range) is something that you need to address with other forms of benchmarking.

But for everything in the right range (milliseconds, seconds, minutes or above), hyperfine is well suited.

Re: Hyperfine: A command-line benchmarking tool

#19

Post by sharkdp »

forrestthewoods wrote: post #7
anotherhue wrote: post #6

Earlier quoted context omitted.

It spawns a new process each time right? I would think that would but a cap on how accurate it can get. For my purposes I use it all the time though, quick and easy sanity-check.

The issue is it runs a kajillion tests to try and be “statistical”. But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”. It’s very much designed for nanosecond to low microsecond benchmarks. Trying to fight this is trying to smash a square peg through a round hole.

> The issue is it runs a kajillion tests to try and be “statistical”.

If you see any reason for putting “statistical” in quotes, please let us know. hyperfine does not run a lot of tests, but it does try to find outliers in your measurements. This is really valuable in some cases. For example: we can detect when the first run of your program takes much longer than the rest of the runs. We can then show you a warning to let you know that you probably want to either use some warmup runs, or a "--prepare" command to clean (OS) caches if you want a cold-cache benchmark.

> But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”.

What is the "best answer you can"?

> It’s very much designed for nanosecond to low microsecond benchmarks.

Absolutely not. With hyperfine, you can not measure execution times in the "low microsecond" range, let alone nanosecond range. See also my other comment.

Re: Hyperfine: A command-line benchmarking tool

#20

Post by oguz-ismail »

sharkdp wrote: post #17
PhilipRoman wrote: post #10

Earlier quoted context omitted.

The shell can be disabled, leaving just fork+exec

Yes. If you don't make use of shell builtins/syntax, you can use hyperfine's `--shell=none`/`-N` option to disable the intermediate shell.

You still need to quote the command though. `hyperfine -N ls "$dir"' won't work, you need `hyperfine -N "ls ${dir@Q}"' or something. It'd be better if you could specify commands like with `find -exec'.
Post reply on HN