Are your memory-bound benchmarking timings normally distributed?

> Consider a sequence of events, with the waiting time for each event being an exponential distribution with rate β

Why is the exponential distribution of waiting times a reasonable assumption in this case?

Usually I don't do any statistical tests when benchmarking and optimizing.

("If your result depends on statistics then you need a better experiment.” Oft quoted remark attributed to Rutherford.)

s/experiment/optimization/

I'm happy with min, max and average. max is actually often the most important.

On the other hand the point of view in this paper is interesting:

"Violating the normality assumption may be the lesser of two evils" (1)

If you really feel the urge to do statistical tests then do it.

(I do not work in medicine or anything dangerous at all)

https://link.springer.com/article/10.3758/s13428-021-01587-5 (1)

In the general case? Because the exponential distribution is directly the result of a poisson process, and poisson processes are also naturally ubiquitous. A poisson process results in poisson distributed counts of events over fixed periods of time, and exponentially distributed times between events. They are used to model everything from rainfall to particle physics, and those concepts do fit very neatly with things that are obviously more complicated like human dynamics.

In the case of memory-bound code benchmarking? Because memory access in NUMA architectures with tiered caches are approximately exponential: L1 vs L2 vs L3 vs main memory vs swap are all exponential increases in latencies. It's obviously more of a step function than a continuous function but the relationship is definitely exponential.

The pages for the exponential distribution and poisson distribution are also informative and interesting.

https://en.wikipedia.org/wiki/Exponential_distribution#Occur...

https://en.wikipedia.org/wiki/Poisson_distribution#Occurrenc...

When I go through the process of finding an appropriate distribution, I usually first look through wikipedia to see if I can find conceptual parallels, just like this. That doesn't mean I settle on it..I'll usually test the distribution fit with functions like those found in fitdistr plus, as well as test to see if residuals are roughly normally distributed.

One hard and fast rule that I use though is that I never use an unbounded distribution to model a bounded process. For compute times, you can't have negative latencies, and you can't have zero latencies, but you can have infinite latencies. Therefore, I would only consider using distributions in the space of (0,infinity). The log normal distribution fits this and is probably adequate here, but the gamma distribution might be a smidge better.

https://cran.r-project.org/web/packages/fitdistrplus/vignett...

I think it’s worthwhile to distinguish between throughput and latency for these sorts of discussions, rather than just talking about performance since scratchpads are usually better for latency (even best-case latency) and caches are usually better for throughput. Though of course in this as in any sort of discussion of computer performance, caveats abound.

The comment is a bit abrupt.

The article questions the normal distribution assumption for benchmark timings.

It says that this assumption would be important for the "standard error of the mean":

"Your error should go down as the square root of the number of measures."

Standard distribution is not even necessary for this:

https://en.wikipedia.org/wiki/Standard_error

However, normal distributions actually behave benignly and are a prerequisite for some statistical tests.

This is where my first comment gets in, that for my benchmarking statistical tests are not necessary at all ...

---

The author writes that he only measures minimum and average, but the maximum values are of course also important and worth to measure.

The author's key point is that without a normal distribution, the maximum values can be highly scattered. I agree and basically it is always nice to know the distribution of the random variable.