Earlier quoted context omitted.
I find k6 a lot nicer for HTTP benching, and no slower to set up than hyperfine (which I love for CLI benching): https://k6.io/
Could hyperfine running curl be an alternative?
That is what I do in my blog post.
Hyperfine: A command-line benchmarking tool
51–56 of 56 posts
Re: Hyperfine: A command-line benchmarking tool
#52Perhaps interesting (for some) to note that hyperfine is from the same author as at least a few other "ne{w,xt} generation" command line tools (that could maybe be seen as part of "rewrite it in Rust", but I don't want to paint the author with a brush they disagree with!!): fd (find alternative; https://github.com/sharkdp/fd ), bat ("supercharged version of the cat command"; https://github.com/sharkdp/bat ), and hexy…
++ to `fd` It’s absolutely my preferred `find` replacement. Its CLI interface just clicks for me and I can quickly express my desires. Quite unlike `find`. `fd` is one of the first packages I install on a new system.
The "funny" thing for me about `fd` is that the set of operations I use for `find` are very hard-wired into my muscle memory from using it for 20+ years, so when I reach for `fd` I often have to reference the man page! I'm getting a little better from more exposure, but it's just different enough from `find` to create a bit of an uncanny valley effect (I think that's the right use of the term...).
Even with that I reach for `fd` for some of its quality-of-life features: respecting .gitignore, its speed, regex-ability. (Though not its choices with color; I am a pretty staunch "--color never" person, for better or worse!)
Anyway, that actually points to another good thing about sharkdp's tools: they have good man pages!!
Re: Hyperfine: A command-line benchmarking tool
#53Earlier quoted context omitted.
I've found pretty good results with the System Trace template in xcode instruments. You can also stack instruments, for example combining the file inspector with a virtual memory inspector. I've run into some memory corruption with it sometimes, though, so be wary of that. Emerge tools has an alternative for iOS at least, maybe one day they'll port it to mac.
I never tried xcode instruments. Is the UX good for this kind of tool?
System trace is pretty comprehensive, as long as it's something in there and you don't need a crazy strong profiler and are okay with occasional hangs in the profiler UI, it's pretty good
Re: Hyperfine: A command-line benchmarking tool
#54Earlier quoted context omitted.
++ to `fd` It’s absolutely my preferred `find` replacement. Its CLI interface just clicks for me and I can quickly express my desires. Quite unlike `find`. `fd` is one of the first packages I install on a new system.
The "funny" thing for me about `fd` is that the set of operations I use for `find` are very hard-wired into my muscle memory from using it for 20+ years, so when I reach for `fd` I often have to reference the man page! I'm getting a little better from more exposure, but it's just different enough from `find` to create a bit of an uncanny valley effect (I think that's the right use of the term...). Even with that I re…
Same here -- I like `fd` for quick stuff, but routinely have to fall back to `find`.
Re: Hyperfine: A command-line benchmarking tool
#55The comment about statistics that I wanted to reply to has disappeared. That commenter said:
> I stand firm in my belief that unless you can prove how CLT applies to your input distributions, you should not assume normality. And if you don't know what you are doing, stop reporting means.
I agree. My research group stopped using Hyperfine because it ranks benchmarked commands by mean, and provides standard deviation as a substitute for a confidence measure. These are not appropriate for heavy-tailed, skewed, and otherwise non-normal distributions.
It's easy to demonstrate that most empirical runtime distributions are not normal. I wrote BestGuess [0] because we needed a better benchmarking tool. Its analysis provides measures of skew, kurtosis, and Anderson-Darling distance from normal, so that you can see how normal or not is your distribution. It ranks benchmark results using non-parametric methods. And, unlike many tools, it saves all of the raw data, making it easy to re-analyze later.
My team also discovered that Hyperfine's measurements are a bit off. It reports longer run times than other tools, including BestGuess. I believe this is due to the approach, which is to call getrusage(), then fork/exec the program to be measured, then call getrusage() again. The difference in user and system times is reported as the time used by the benchmarked command, but unfortunately this time also includes cycles spent in the Rust code for managing processes (after the fork but before the exec).
BestGuess avoids external libraries (we can see all the relevant code), does almost nothing after the fork, and uses wait4() to get measurements. The one call to wait4() gives us what the OS measured by its own accounting for the benchmarked command.
While BestGuess is still a work in progress (not yet at version 1.0), my team has started using it regularly. I plan to continue its development, and I'll write it up soon at [1].
[0] https://gitlab.com/JamieTheRiveter/bestguess [1] https://jamiejennings.com
Re: Hyperfine: A command-line benchmarking tool
#56What database product does the community commonly send benchmark results to? This tool is great, but I'd love to analyze results relationally.
You could export data to csv and do relational analytics with DuckDB quite easily.