RAM is the new disk – and how to measure its performance (2015) • Hacker News Board

Re: RAM is the new disk – and how to measure its performance (2015)

#31

Post by WJW » Thu, Jan 21, 2021, 10:27 PM UTC

BenoitP wrote: post #29

vlovich123 wrote: post #2

I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…

> HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU Well, that seems to be the exact definition of what UPMEM is doing: https://www.upmem.com/upmem-announces-silicon-based-processi... Between the M1, GPUs, TPUs, RISC-V interesting times are coming in hardware. I blame the physical limits, which are putting the Duke N…

You'd hope that the only way to better performance was HW specialisation, but there are SO MANY algorithmic improvements still to be made. Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. In a fairly popular open source library too.

There's still loads of performance to be gained just by being better programmers.

Re: RAM is the new disk – and how to measure its performance (2015)

#32

Post by jzwinck » Thu, Jan 21, 2021, 10:29 PM UTC

fctorial wrote: post #22

cogman10 wrote: post #6

Earlier quoted context omitted.

Interestingly enough, this is why Simultaneous multithreading [1] exists! The revelation that "The CPU could be doing useful work while stalled waiting on a load" has lead to CPU designers "faking" the number of cores available to the OS to allow the CPU to do more useful work while different "threads" are paused waiting on memory to come back with the data they need. [1] https://en.wikipedia.org/wiki/Simultaneous_mu…

They aren't faking the number. A hyperthreaded core is two separate cores that share alu, fpu etc.

This is not how Intel sees it. They describe hyperthreading as a single core having two (or more, but usually two) sets of architectural state which basically means registers. It's not two cores, it is one core that can switch between two different instruction pointers. They share almost everything else apart from the APIC.

Re: RAM is the new disk – and how to measure its performance (2015)

#33

Post by freeone3000 » Thu, Jan 21, 2021, 10:29 PM UTC

dragontamer wrote: post #30

ben509 wrote: post #27

Earlier quoted context omitted.

Well, all memory operations must be translated by an MMU. So can the MMU be a distributed beast that lives in the CPU, the memory and the IO?

> So can the MMU be a distributed beast that lives in the CPU, the memory and the IO? That "blah = blah->next" memory operation could be: * Going to swap thanks to swap / overcommit behavior on Linux/Windows. * Or going to a file thanks to mmap * Going to Ethernet and to another computer thanks to RDMA So... no. I'm pretty sure the CPU is the proper place for that kind of routing.

why is that? we need the CPU to handle page fault interrupts, in order to populate the RAM. But assuming the page is already in RAM, there's no reason any of the memory accesses actually need to go through the CPU. (hardware can already raise interrupts; if the MMU can raise a page fault indicator, then you might be able to bypass CPU entirely until a new page needs to be loaded)

moreover, if we have support for mmap at the MMU level, we can cut the CPU bottleneck for disk access entirely. the disk controller can already handle DMA, but there's simply no way for things that aren't the CPU to trigger it. DirectStorage is an effort for GPUs to trigger it, but what if we could also trigger it by other means?

Re: RAM is the new disk – and how to measure its performance (2015)

#34

Post by Arelius » Thu, Jan 21, 2021, 10:30 PM UTC

WJW wrote: post #31

BenoitP wrote: post #29

Earlier quoted context omitted.

> HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU Well, that seems to be the exact definition of what UPMEM is doing: https://www.upmem.com/upmem-announces-silicon-based-processi... Between the M1, GPUs, TPUs, RISC-V interesting times are coming in hardware. I blame the physical limits, which are putting the Duke N…

You'd hope that the only way to better performance was HW specialisation, but there are SO MANY algorithmic improvements still to be made. Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. In a fairly popular open source library too. There's still loads of performance to be gained just by being better programmers.

> Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap.

I feel like this story needs an ending? Did somebody re-implement it using a heap and found significant performance wins? Or was the sorted array used on purpose to take advantage of some specific cache constraints and actually end up being a huge win?

Re: RAM is the new disk – and how to measure its performance (2015)

#35

Post by spockz » Thu, Jan 21, 2021, 10:42 PM UTC

WJW wrote: post #31

BenoitP wrote: post #29

Earlier quoted context omitted.

> HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU Well, that seems to be the exact definition of what UPMEM is doing: https://www.upmem.com/upmem-announces-silicon-based-processi... Between the M1, GPUs, TPUs, RISC-V interesting times are coming in hardware. I blame the physical limits, which are putting the Duke N…

You'd hope that the only way to better performance was HW specialisation, but there are SO MANY algorithmic improvements still to be made. Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. In a fairly popular open source library too. There's still loads of performance to be gained just by being better programmers.

How do you propose to fix this? Should languages include high(er?) performance data structures in their standard libraries? Or possibly even include some segmentation for small/medium/huge data sets?

Re: RAM is the new disk – and how to measure its performance (2015)

#36

Post by jeffbee » Thu, Jan 21, 2021, 10:49 PM UTC

freeone3000 wrote: post #33

dragontamer wrote: post #30

Earlier quoted context omitted.

> So can the MMU be a distributed beast that lives in the CPU, the memory and the IO? That "blah = blah->next" memory operation could be: * Going to swap thanks to swap / overcommit behavior on Linux/Windows. * Or going to a file thanks to mmap * Going to Ethernet and to another computer thanks to RDMA So... no. I'm pretty sure the CPU is the proper place for that kind of routing.

why is that? we need the CPU to handle page fault interrupts, in order to populate the RAM. But assuming the page is already in RAM, there's no reason any of the memory accesses actually need to go through the CPU. (hardware can already raise interrupts; if the MMU can raise a page fault indicator, then you might be able to bypass CPU entirely until a new page needs to be loaded) moreover, if we have support for mmap…

Pointers are virtual addresses but memory is accessed physically. All of the means of translating virtual to physical are in the CPU. If you are proposing throwing out virtual addressing, I imagine you won't get a lot of support for that idea.

Re: RAM is the new disk – and how to measure its performance (2015)

#37

Post by charlesdaniels » Thu, Jan 21, 2021, 10:51 PM UTC

vlovich123 wrote: post #2

I think how DMA operates needs another serious look. Right now we have to fetch everything into the CPU before we can make decisions. What if we had asynchronous HW embedded within the memory that could be given a small (safe) executable program to process the memory in-place rather than evaluating it on the CPU. In other words, a linked list would be much faster and simpler to traverse. A lot of the software archite…

This is indeed an idea that has been coming up from time to time for over 25 years at least. I think the earliest publication in this space was Gockhale's Terasys (DOI 10.1109/2.375174).

It is a good idea in principle, but it's never really taken off to my knowledge. Parallel programming is hard. Deeply embedded programming is hard. The languages we have are mostly bad at both.

If you want to search for more, the keyword is "processor in memory" or "processing in memory" (PIM). Also "in memory computing" is commonly used as well. A number of groups are working on this right now. The new buss is "memristors" (memristive computing). Whether or not any of it actually ends up working or being commercially viable outside of a lab remains to be seen.

Re: RAM is the new disk – and how to measure its performance (2015)

#38

Post by dragontamer » Thu, Jan 21, 2021, 11:00 PM UTC

freeone3000 wrote: post #33

dragontamer wrote: post #30

Earlier quoted context omitted.

> So can the MMU be a distributed beast that lives in the CPU, the memory and the IO? That "blah = blah->next" memory operation could be: * Going to swap thanks to swap / overcommit behavior on Linux/Windows. * Or going to a file thanks to mmap * Going to Ethernet and to another computer thanks to RDMA So... no. I'm pretty sure the CPU is the proper place for that kind of routing.

why is that? we need the CPU to handle page fault interrupts, in order to populate the RAM. But assuming the page is already in RAM, there's no reason any of the memory accesses actually need to go through the CPU. (hardware can already raise interrupts; if the MMU can raise a page fault indicator, then you might be able to bypass CPU entirely until a new page needs to be loaded) moreover, if we have support for mmap…

Okay, lets say page-faults are off the table for some reason. Lets think about what can happen even if everything is in RAM still.

* MMAP is still on the table: different processes can share RAM at different addresses. (Process#1 thinks the data is at memory location 0x90000000, Process#2 thinks the data is at 0x70000000, but in both cases, the data is at physical location 0x42).

* Physical location 0x42 is a far-read on a far-away NUMA node. Which means the CPU#0 now needs to send a message to a CPU#1 very far away to get a copy of that RAM. This message traverses Intel Ultrapath Interconnect or AMD Infinity fabric (proprietary details), but its a remote message that happens nonetheless.

* Turns out CPU#1 has modified location 0x42. Now CPU#1 must push the most recent copy out of L1 cache, into L2 cache... then into L3 cache, and then send it back to CPU#0. CPU#0 has to wait until this process is done. If CPU#1 wants to modify the data again (or even read it), it may require messages from CPU#0 (who is now the owner of the data, according to simple MESI models).

Modern computers work very hard to hold the illusion of a singular memory space. Eventually, these details are turned into a consistent memory model and become well-ordered sequential operations. The CPU is a good place for that.

---------------

That's how stuff works _today_. If you wanted to make a new programming model that's incompatible, that's fine. (CUDA does it: GPUs don't have as many virtual-memory features as a CPU. And __shared__ memory has a different model than L1 cache.)

But if you invent a new memory model that does things differently, it means that it won't work for the vast majority of code. Which means you need to bootstrap a new programming environment (much like how CUDA bootstrapped a new community from scratch).

Re: RAM is the new disk – and how to measure its performance (2015)

#39

Post by WJW » Thu, Jan 21, 2021, 11:05 PM UTC

Arelius wrote: post #34

WJW wrote: post #31

Earlier quoted context omitted.

You'd hope that the only way to better performance was HW specialisation, but there are SO MANY algorithmic improvements still to be made. Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. In a fairly popular open source library too. There's still loads of performance to be gained just by being better programmers.

> Just the other day I found a case someone had rolled their own priority queue with a sorted array instead of a heap. I feel like this story needs an ending? Did somebody re-implement it using a heap and found significant performance wins? Or was the sorted array used on purpose to take advantage of some specific cache constraints and actually end up being a huge win?

The story ends (so far) with me filing a GH issue and offering a PR, my own benchmarking getting an average 3x speedup over a wide range of inputs. I happen to know that the maintainer has been on vacation for the last week and only got back a few days ago so no further discussion has been had yet.

Re: RAM is the new disk – and how to measure its performance (2015)

#40

Post by dragontamer » Thu, Jan 21, 2021, 11:12 PM UTC

jeffbee wrote: post #13

geocar wrote: post #12

Earlier quoted context omitted.

Linked-list chasing (as in vlovich123's example) isn't bandwidth-limited as much as it is latency: SMT helps here because you're able to enqueue multiple wait-states "simultaneously".

Anything you can do with SMT you can do without SMT using ILP instead. SMT does not grant a CPU extra cache-filling resources.

Depends on the implementation of SMT.

I know that POWER9 for example, reserves the superslices for different threads as you go from SMT1 (1-thread per core) up to SMT4 (4-threads per core).

Thread#0 and Thread#1 get Slice0, while Thread#2 and Thread#3 get Slice1.

--------

For an even more extreme example, Bulldozer's implementation of SMT / Cores / whatever you wanna call it... the L1 cache was split between the two cores. So you'd rather have 2-threads doing two different things (even if they shared the same decoder and significantly shared the same resources), because you'd "magically" get access to more L1 cache.

Even on modern Skylake / Zen systems, some SMT resources are locked to one thread or the other. So you do in fact get more resources from SMT.

On the other hand: you're absolutely right in that pointer-chasing can be done in parallel in a single thread due to ILP (and that ILP is probably even preferable to using more threads). Still, the fact that SMT systems are implemented so differently and so weirdly at times... it means that we can't really make general rules about SMT systems.