21–30 of 50 posts
One thing I discovered benching Oracle instances is that cloud performance is extremely non deterministic. Sometimes you get variance each run. Sometimes you get consistent results, but then you bench it later and the results are consistent, but totally different. Then I would run the same bench on my laptop and (with turbo disabled) ands its pretty much always consistent. I can only guess why... turbo on the host cp…
I feel obligated to point out that, by the central limit theorem, it doesn’t matter if the timings are normally distributed. The measurements only need to be independently and identically distributed with finite variance for the estimate of the mean to be normally distributed. https://en.m.wikipedia.org/wiki/Central_limit_theorem
The other point he makes is that the mean is not very representative in this wide log-normal scenario, and optimizing for the difference between min and mean is more relevant.
My 2 cents.
Also to nitpick on the article:
"It is not possible, in a normal distribution, to be multiple times the standard deviation away from the mean."
I guess he meant probable :)
One thing I discovered benching Oracle instances is that cloud performance is extremely non deterministic. Sometimes you get variance each run. Sometimes you get consistent results, but then you bench it later and the results are consistent, but totally different. Then I would run the same bench on my laptop and (with turbo disabled) ands its pretty much always consistent. I can only guess why... turbo on the host cp…
You're sharing the same hardware, with who knows how many others.
Bare metal ia OK if you must test NUMA and such, but you are still at the mercy of turbo behavior, where you can completely control that in a local machine.
You might want to check out the gamma distribution. It is also zero-bounded just like the log normal distribution, but it was originally created to model waiting times within queue theory, which is actually an excellent parallel to the idea of measuring compute latency.
Gamma and delta distributions have been very helpful to me in performance work, as well as non-parametric statistical tests. However, when you try to tell a lot of other engineers about them, they don't really understand why a t-test and a standard deviation doesn't work.
Is "possible" supposed to be "probable" there?
I've been writing micro memory benchmarks and have been rather surprised how hard something as simple as quantifying latency and bandwidth under multicore loads can be. The memory hierarchy is getting ever more complex. Cacheline sizes, prefetch, 3 levels of cache, TLB effects, page alignments, cache associativity, etc. Also have to be careful that the compiler doesn't optimize away parts of your code. It's quite tri…
I want a viewport onto a parallel universe where we embraced NUMA more thoroughly, and instead of making processors with ever growing layers of transparent caching, we just put 16MB of working memory on each chip. Partitionable for concurrent workloads. Maybe it could never have been then, but maybe it can be now, or soon. For instance a borrow checker might be a very good way to help decide how and when to move work…
[1] https://en.m.wikipedia.org/wiki/Cell_(processor)#Synergistic...
I've found timings in general were lognormal a long time ago. I had to routinely apply transformations to get meaningful ANOVA and DOE results.
I feel obligated to point out that, by the central limit theorem, it doesn’t matter if the timings are normally distributed. The measurements only need to be independently and identically distributed with finite variance for the estimate of the mean to be normally distributed. https://en.m.wikipedia.org/wiki/Central_limit_theorem
I think he's saying that since the distribution follows a log-normal, the mean is harder to pin down than if it were a normal due to the higher variance. From the CLT, the variance of the mean statistic is var(X)/sqrt(N) -- the var(X) is higher and he'd expect from an assumed normal of X. The other point he makes is that the mean is not very representative in this wide log-normal scenario, and optimizing for the diff…