[dead]
Samsung's Processing-in-Memory (PIM)
81–90 of 120 posts
Re: Samsung's Processing-in-Memory (PIM)
#82Earlier quoted context omitted.
As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required. Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
> You could run MACs directly in RAM Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
Re: Samsung's Processing-in-Memory (PIM)
#83Earlier quoted context omitted.
> Whilst processing in memory is clearly the future How clear is that? The idea has been around for about 60 years, and many attempts made by people who thought the same thing. Maybe this time it'll be the future.
Because now the usecase fits the needs.
Maybe. The use cases have always been around FP arithmetic over arrays, because that's what is easy to parallelize. I saw a standalone systolic array box attached to a MicroVAX circa 1990. Lots of PIM approaches in the mid-90s, too, but mostly what survived from that era are shared memory NUMA multiprocessors and using GPUs for general purpose computing.
Re: Samsung's Processing-in-Memory (PIM)
#84Earlier quoted context omitted.
> You could run MACs directly in RAM Sure, MACs are nice. However, unless there other, PIM-specific/optimal, algorithms, regular matrix multiplication algorithms like tiling-based won’t work here I think — how would the tile be shared? By doing read/write all the time?
Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes?
Also, doesn’t it mean that you forgo batching?
Re: Samsung's Processing-in-Memory (PIM)
#85Earlier quoted context omitted.
There are fundamental issues here and I think the article only touched on a few. On the software side this completely blows up the whole virtual memory concept. We will need different operating systems.
why would it? the parent OS can already handle physically contiguous allocations so these should be no different (with the exception that a separate interface can be used to do compute over these buffers/pages).
If it requires physically contiguous RAM to work, then it's not really participating in the full virtual memory system, really. It would be using an exception to it, that can be accommodated to some extent by the OS, but not sitting in demand-paged storage like the rest of the system.
Re: Samsung's Processing-in-Memory (PIM)
#86Earlier quoted context omitted.
Yeah this sounds a lot like a natural evolution of SIMD for me, just cut out the middleman and put the SIMD units straight into RAM. I can imagine some power savings for always-on display applications too. Rather than periodically waking the CPU/GPU to update the frame buffer, you can just stash small bits of periodic logic in memory (e.g. move the second hand of a clock).
I'm having a hard time imagining this SIMD replacement except for extremely narrow use cases. Are you suggesting the PIM would have a full blown IO controller and cache subsystem to fetch remote operands? I assume PIM is only going to work well for chunky streaming over the data within that particular memory module. Something that address and operate on whole rows at once and has minimal buffering between the RAM acc…
> PIM would have a full blown IO controller and cache subsystem to fetch remote operands
Don't we already have those in mainstream computing in the form of dedicated silicon in DMA controllers? Programmed input–output performance is often low throughput, high jitter and uses a lot of CPU.
Re: Samsung's Processing-in-Memory (PIM)
#87Earlier quoted context omitted.
I'm having a hard time imagining this SIMD replacement except for extremely narrow use cases. Are you suggesting the PIM would have a full blown IO controller and cache subsystem to fetch remote operands? I assume PIM is only going to work well for chunky streaming over the data within that particular memory module. Something that address and operate on whole rows at once and has minimal buffering between the RAM acc…
> PIM would have a full blown IO controller and cache subsystem to fetch remote operands Don't we already have those in mainstream computing in the form of dedicated silicon in DMA controllers? Programmed input–output performance is often low throughput, high jitter and uses a lot of CPU.
Yes, but a DMA controller sit out on the memory bus to do the same kind of work the CPU could be doing, controlling bus traffic between memory modules.
I think the whole idea of PIM is to be able to scale up and do work locally within the memory module, without the bottleneck of the system memory bus. This only works for embarrassingly parallel workloads that don't actually depend on the bisection bandwidth across the global memory.
If you start talking about PIM that is all interconnected, your application is back to being bound by the system memory bus. Maybe it's a new package design, but you're basically building yet another NUMA interconnect architecture, not a PIM architecture.
Re: Samsung's Processing-in-Memory (PIM)
#88[flagged]
Re: Samsung's Processing-in-Memory (PIM)
#89Earlier quoted context omitted.
Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes? Also, doesn’t it mean that you forgo batching?
[deleted]
Re: Samsung's Processing-in-Memory (PIM)
#90The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…
I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module. Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region. Let's say you would need to run a sum over megabytes of data like for accounting. You…
If Infiniband does this for MPI on the network and realizes Sun's "The network is the computer" dream, I believe we can do this for other parts of the hardware, as well. Not only for AI, HPC will love this idea.