Hyperfine: A command-line benchmarking tool
1–10 of 56 posts
Re: Hyperfine: A command-line benchmarking tool
#2Hyperfine is a great tool but when I was using it at Deno to benchmark startup time there was a lot of weirdness around the operating system apparently caching inodes of executables.
If you are looking at shaving sub 20ms numbers, be aware you may need to pull tricks on macos especially to get real numbers.
Re: Hyperfine: A command-line benchmarking tool
#3What database product does the community commonly send benchmark results to? This tool is great, but I'd love to analyze results relationally.
Re: Hyperfine: A command-line benchmarking tool
#4Hyperfine is a great tool but when I was using it at Deno to benchmark startup time there was a lot of weirdness around the operating system apparently caching inodes of executables. If you are looking at shaving sub 20ms numbers, be aware you may need to pull tricks on macos especially to get real numbers.
I've found pretty good results with the System Trace template in xcode instruments. You can also stack instruments, for example combining the file inspector with a virtual memory inspector.
I've run into some memory corruption with it sometimes, though, so be wary of that. Emerge tools has an alternative for iOS at least, maybe one day they'll port it to mac.
Re: Hyperfine: A command-line benchmarking tool
#5Hyperfine is hyper frustrating because it only works with really really fine microsecond level benchmarks. Once you get into the millisecond range it’s worthless.
Re: Hyperfine: A command-line benchmarking tool
#6Hyperfine is hyper frustrating because it only works with really really fine microsecond level benchmarks. Once you get into the millisecond range it’s worthless.
It spawns a new process each time right? I would think that would but a cap on how accurate it can get.
For my purposes I use it all the time though, quick and easy sanity-check.
Re: Hyperfine: A command-line benchmarking tool
#7Hyperfine is hyper frustrating because it only works with really really fine microsecond level benchmarks. Once you get into the millisecond range it’s worthless.
It spawns a new process each time right? I would think that would but a cap on how accurate it can get. For my purposes I use it all the time though, quick and easy sanity-check.
The issue is it runs a kajillion tests to try and be “statistical”. But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”. It’s very much designed for nanosecond to low microsecond benchmarks. Trying to fight this is trying to smash a square peg through a round hole.
Re: Hyperfine: A command-line benchmarking tool
#8Hyperfine is hyper frustrating because it only works with really really fine microsecond level benchmarks. Once you get into the millisecond range it’s worthless.
It spawns a new process each time right? I would think that would but a cap on how accurate it can get. For my purposes I use it all the time though, quick and easy sanity-check.
It spawns a new shell for each run and subtracts the average shell startup time from final results. Too much noise
Re: Hyperfine: A command-line benchmarking tool
#9Earlier quoted context omitted.
It spawns a new process each time right? I would think that would but a cap on how accurate it can get. For my purposes I use it all the time though, quick and easy sanity-check.
The issue is it runs a kajillion tests to try and be “statistical”. But there’s no good way to say “just run it for 5 seconds and give me the best answer you can”. It’s very much designed for nanosecond to low microsecond benchmarks. Trying to fight this is trying to smash a square peg through a round hole.
At least it gives some numbers and point in a direction:
$ hyperfine --warmup 3 './hello-world-bin-sh.sh' './hello-world-env-python3.py'
Benchmark 1: ./hello-world-bin-sh.sh
Time (mean ± σ): 1.3 ms ± 0.4 ms [User: 1.0 ms, System: 0.5 ms]
...
Benchmark 2: ./hello-world-env-python3.py
Time (mean ± σ): 43.1 ms ± 1.4 ms [User: 33.6 ms, System: 8.4 ms]
...Re: Hyperfine: A command-line benchmarking tool
#10Earlier quoted context omitted.
It spawns a new process each time right? I would think that would but a cap on how accurate it can get. For my purposes I use it all the time though, quick and easy sanity-check.
It spawns a new shell for each run and subtracts the average shell startup time from final results. Too much noise
The shell can be disabled, leaving just fork+exec