Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are your memory-bound benchmarking timings normally distributed?

lemire.me

Post reply on HN
Print view View original

11–20 of 50 posts

Re: Are your memory-bound benchmarking timings normally distributed?

#11

Post by kurthr »

ericpauley wrote: post #2

Great article! Statistical fallacies are rampant in performance eval, even in academic settings. When designing statistical tests for performance, the keyword you want to use here is non-parametric. I.e., a U-test is a non-parametric analog to the t-test. It just looks at the rank statistics of results instead of their value, thus eliminating dependence on the underling distribution. Another issue that pops up is sam…

Many would do well to look at simple DOE when building experimental design. At least look at the data distributions before taking means and stdev.

   NIST has some nice simple descriptions and example experimental designs:
   https://www.itl.nist.gov/div898/handbook/pmd/pmd.htm

Re: Are your memory-bound benchmarking timings normally distributed?

#12

Post by sliken »

hinkley wrote: post #8
sliken wrote: post #5

I've been writing micro memory benchmarks and have been rather surprised how hard something as simple as quantifying latency and bandwidth under multicore loads can be. The memory hierarchy is getting ever more complex. Cacheline sizes, prefetch, 3 levels of cache, TLB effects, page alignments, cache associativity, etc. Also have to be careful that the compiler doesn't optimize away parts of your code. It's quite tri…

I want a viewport onto a parallel universe where we embraced NUMA more thoroughly, and instead of making processors with ever growing layers of transparent caching, we just put 16MB of working memory on each chip. Partitionable for concurrent workloads. Maybe it could never have been then, but maybe it can be now, or soon. For instance a borrow checker might be a very good way to help decide how and when to move work…

Well the tools are there, and the hardware is there as well.

Early AMD epycs had multiple chiplets, each with it's own memory controller. However that means that even on a single socket different chiplets would see different memory latencies and some apps/benchmarks people cared about handled it poorly.

The newest AMD Epycs have a unified memory controller per socket, so all chiplets see the same memory latency. But now Intel entered the chiplet game, and now has one memory controller per chiplet again.

There's plenty of support for NUMA support in the kernel and various language runtimes. Apps and OSs can pin processes to cores, allocate local ram, and migrate non-local ram to local or migrate processes to cores that are memory local. Generally the automatic stuff works reasonably, but numactl and various NUMA related calls let you override those decisions as needed.

Re: Are your memory-bound benchmarking timings normally distributed?

#13

Post by hinkley »

sliken wrote: post #12
hinkley wrote: post #8

Earlier quoted context omitted.

I want a viewport onto a parallel universe where we embraced NUMA more thoroughly, and instead of making processors with ever growing layers of transparent caching, we just put 16MB of working memory on each chip. Partitionable for concurrent workloads. Maybe it could never have been then, but maybe it can be now, or soon. For instance a borrow checker might be a very good way to help decide how and when to move work…

Well the tools are there, and the hardware is there as well. Early AMD epycs had multiple chiplets, each with it's own memory controller. However that means that even on a single socket different chiplets would see different memory latencies and some apps/benchmarks people cared about handled it poorly. The newest AMD Epycs have a unified memory controller per socket, so all chiplets see the same memory latency. But…

I missed the 16 bit era by dragging my feet and then leaping onto 32 bit as soon as it was viable.

64k of memory was not really enough to do useful things and it occupied so much of your design. But 4GB is a pretty big leap from 64k. There's lots of interesting things a person can do with 10, 30, 100MB of memory, without having to treat every piece of data like it sits in a perfectly flat data plane. And with 64 bit addressing you have ample, ample space to use some bits as metadata that lets you know things about the memory without dereferencing it first.

Re: Are your memory-bound benchmarking timings normally distributed?

#15

Post by sliken »

hinkley wrote: post #13
sliken wrote: post #12

Earlier quoted context omitted.

Well the tools are there, and the hardware is there as well. Early AMD epycs had multiple chiplets, each with it's own memory controller. However that means that even on a single socket different chiplets would see different memory latencies and some apps/benchmarks people cared about handled it poorly. The newest AMD Epycs have a unified memory controller per socket, so all chiplets see the same memory latency. But…

I missed the 16 bit era by dragging my feet and then leaping onto 32 bit as soon as it was viable. 64k of memory was not really enough to do useful things and it occupied so much of your design. But 4GB is a pretty big leap from 64k. There's lots of interesting things a person can do with 10, 30, 100MB of memory, without having to treat every piece of data like it sits in a perfectly flat data plane. And with 64 bit…

Heh, I was running DOS and Deskview, windows 3 was out of coming soon. I jumped into Linux with a 386sx-16, not even a FPU. I think it had 4MB ram, eventually upgraded to 16mb.

Your ideas remind me of the Sun MAJC CPU. Claimed to be designed to run Java well, be aware of Java Objects, enabling prefetch of objects, speculative execution without side effects, and other pointer magic related to assuming JVM+Java instead of C/C++. Sounded really promising, sadly died shortly afterwards. An expert I talked with later mentioned that it was all marketing and it was just a normal CPU for it's time. Sad. Does seem like today's new languages could result in a CPU with significantly more freedom for speculation, optimization, and efficiency if it assumed a particular language runtime.

Re: Are your memory-bound benchmarking timings normally distributed?

#16

Post by hinkley »

sliken wrote: post #15
hinkley wrote: post #13

Earlier quoted context omitted.

I missed the 16 bit era by dragging my feet and then leaping onto 32 bit as soon as it was viable. 64k of memory was not really enough to do useful things and it occupied so much of your design. But 4GB is a pretty big leap from 64k. There's lots of interesting things a person can do with 10, 30, 100MB of memory, without having to treat every piece of data like it sits in a perfectly flat data plane. And with 64 bit…

Heh, I was running DOS and Deskview, windows 3 was out of coming soon. I jumped into Linux with a 386sx-16, not even a FPU. I think it had 4MB ram, eventually upgraded to 16mb. Your ideas remind me of the Sun MAJC CPU. Claimed to be designed to run Java well, be aware of Java Objects, enabling prefetch of objects, speculative execution without side effects, and other pointer magic related to assuming JVM+Java instead…

> Your ideas remind me of the Sun MAJC CPU.

Probably not a coincidence. It was memory of some Sun product data sheets that got me ruminating on this line of thought, though I don't think it was MAJC. 4MB per core, 4 cores per daughter card, if memory serves. Those product lines were probably a big reason why Java had to solve the NUMA memory problem, and then other languages just copied what they did.

Re: Are your memory-bound benchmarking timings normally distributed?

#17

Post by marcosdumay »

sliken wrote: post #12
hinkley wrote: post #8

Earlier quoted context omitted.

I want a viewport onto a parallel universe where we embraced NUMA more thoroughly, and instead of making processors with ever growing layers of transparent caching, we just put 16MB of working memory on each chip. Partitionable for concurrent workloads. Maybe it could never have been then, but maybe it can be now, or soon. For instance a borrow checker might be a very good way to help decide how and when to move work…

Well the tools are there, and the hardware is there as well. Early AMD epycs had multiple chiplets, each with it's own memory controller. However that means that even on a single socket different chiplets would see different memory latencies and some apps/benchmarks people cared about handled it poorly. The newest AMD Epycs have a unified memory controller per socket, so all chiplets see the same memory latency. But…

Hum... The hardware isn't actually there.

It has the capabilities the GP wants, but it doesn't export them in a way you could tune into that NUMA machine.

Re: Are your memory-bound benchmarking timings normally distributed?

#18

Post by sliken »

marcosdumay wrote: post #17
sliken wrote: post #12

Earlier quoted context omitted.

Well the tools are there, and the hardware is there as well. Early AMD epycs had multiple chiplets, each with it's own memory controller. However that means that even on a single socket different chiplets would see different memory latencies and some apps/benchmarks people cared about handled it poorly. The newest AMD Epycs have a unified memory controller per socket, so all chiplets see the same memory latency. But…

Hum... The hardware isn't actually there. It has the capabilities the GP wants, but it doesn't export them in a way you could tune into that NUMA machine.

You can run any process on any core and allocate local memory to that core. What is missing?

Sure you can't hide metadata in the unused bits of a pointer, but that didn't seem particularly critical for making good use of a NUMA machine.

Re: Are your memory-bound benchmarking timings normally distributed?

#19

Post by singron »

hinkley wrote: post #8
sliken wrote: post #5

I've been writing micro memory benchmarks and have been rather surprised how hard something as simple as quantifying latency and bandwidth under multicore loads can be. The memory hierarchy is getting ever more complex. Cacheline sizes, prefetch, 3 levels of cache, TLB effects, page alignments, cache associativity, etc. Also have to be careful that the compiler doesn't optimize away parts of your code. It's quite tri…

I want a viewport onto a parallel universe where we embraced NUMA more thoroughly, and instead of making processors with ever growing layers of transparent caching, we just put 16MB of working memory on each chip. Partitionable for concurrent workloads. Maybe it could never have been then, but maybe it can be now, or soon. For instance a borrow checker might be a very good way to help decide how and when to move work…

This is basically how tile renderers work in mobile GPUs. Supercomputers also have a similar design with MPI and toroid networks although on a multi-machine scale.

Re: Are your memory-bound benchmarking timings normally distributed?

#20

Post by Dylan16807 »

hinkley wrote: post #8
sliken wrote: post #5

I've been writing micro memory benchmarks and have been rather surprised how hard something as simple as quantifying latency and bandwidth under multicore loads can be. The memory hierarchy is getting ever more complex. Cacheline sizes, prefetch, 3 levels of cache, TLB effects, page alignments, cache associativity, etc. Also have to be careful that the compiler doesn't optimize away parts of your code. It's quite tri…

I want a viewport onto a parallel universe where we embraced NUMA more thoroughly, and instead of making processors with ever growing layers of transparent caching, we just put 16MB of working memory on each chip. Partitionable for concurrent workloads. Maybe it could never have been then, but maybe it can be now, or soon. For instance a borrow checker might be a very good way to help decide how and when to move work…

Partitioning memory and cores is easy enough, and a bunch of Xeons have cache partitioning. It's still generally transparent to the program, but I don't think that makes it less NUMA.
Post reply on HN