Samsung's Processing-in-Memory (PIM)
91–100 of 120 posts
Re: Samsung's Processing-in-Memory (PIM)
#92Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…
Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.
With the size of these matrices I don't think they are even meaningfully colocated with themselves in memory. You'll end up with some dataflow TPU architecture anyway because you'll have to stream the second matrix to multiply against.
Re: Samsung's Processing-in-Memory (PIM)
#93The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…
I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module. Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region. Let's say you would need to run a sum over megabytes of data like for accounting. You…
> run a sum over megabytes of data like for accounting
It's been many decades since the last time somebody ran a sum over megabytes of data for accounting and though "damn that's a bottleneck I need to optimize".
> Personally I'm a big fan of the "in-ram accelerator" idea especially for server space.
The operations this model supports are so extremely limited that you would be hard pressed to find applications where it's worth it.
Re: Samsung's Processing-in-Memory (PIM)
#94Earlier quoted context omitted.
why would it? the parent OS can already handle physically contiguous allocations so these should be no different (with the exception that a separate interface can be used to do compute over these buffers/pages).
If it requires physically contiguous RAM to work, then it's not really participating in the full virtual memory system, really. It would be using an exception to it, that can be accommodated to some extent by the OS, but not sitting in demand-paged storage like the rest of the system.
And the security folks will have tons of fun with this enormously leaky abstraction too.
Re: Samsung's Processing-in-Memory (PIM)
#95Earlier quoted context omitted.
why would it? the parent OS can already handle physically contiguous allocations so these should be no different (with the exception that a separate interface can be used to do compute over these buffers/pages).
If it requires physically contiguous RAM to work, then it's not really participating in the full virtual memory system, really. It would be using an exception to it, that can be accommodated to some extent by the OS, but not sitting in demand-paged storage like the rest of the system.
Linux has been dealing with this kind of thing for over a decade now. Specialized SoCs love their memory carveouts.
Re: Samsung's Processing-in-Memory (PIM)
#96The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…
Yes, and no.
Two decades ago it was a challenge to get people to see that what they were doing was preventing horizontal scaling. Today horizontal scaling is table stakes and people don't even always register that they are doing it. It's just how we do things, no thoughts.
PIM requires problems to be decomposed into horizontal scaling problems. Then what you should do with PIM is take a problem that used to be solved by 2 racks of computers and squeeze it down to less than half a rack by stuffing a bunch of these into a single box to do 8-10x as much work per box (and double the cluster size to offset Jevons' Paradox because it's so cheap now that you'll do 2x as much of it)
Re: Samsung's Processing-in-Memory (PIM)
#97Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…
"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of…
If what we want to do with this is make cheap QKV sweeps, then "a weak NPU with a lot of mem bandwidth" seems good enough? Exactly the tool for that job, and nothing else.
Also spares us the trouble of dealing with weights. By the time we're in QKV realm, the weights have already weighted.
Re: Samsung's Processing-in-Memory (PIM)
#98Earlier quoted context omitted.
Attention calculations aren't shared across more than one vector during next token prediction (thinking and writing) which this sounds almost perfect for. Per attention layer, for deepseek at 1M context, you want to broadcast a single 1KB vector to 4GB of dot products, and map reduce a 1KB vector back.
How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes? Also, doesn’t it mean that you forgo batching?
Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.
Re: Samsung's Processing-in-Memory (PIM)
#99The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…
There are fundamental issues here and I think the article only touched on a few. On the software side this completely blows up the whole virtual memory concept. We will need different operating systems.
Maybe PIM will push this forward, but I still think we're doing something fundamentally wrong by not just embracing NUMA and trying to do something Sun tried decades ago, which is have number of cores share 4GB of semiprivate working
memory.
We've kind of half-assed it with DDR memory banks, but it mostly introduces mysterious slowdowns that are difficult to reason about and I think we would be better served I think by making a formal thing. Instead of introducing an L4 cache we could do this instead, and reduce the size of the L1-L3 caches, which shortens lookup time and thus latency.
For legacy apps, you could provide facilities for the OS to 'page' blocks in from main memory, but the speed would come from managing the workload imperatively, starting loads in the background before the data is actually needed, and dumps after it is last touched.
Re: Samsung's Processing-in-Memory (PIM)
#100Earlier quoted context omitted.
why would it? the parent OS can already handle physically contiguous allocations so these should be no different (with the exception that a separate interface can be used to do compute over these buffers/pages).
If it requires physically contiguous RAM to work, then it's not really participating in the full virtual memory system, really. It would be using an exception to it, that can be accommodated to some extent by the OS, but not sitting in demand-paged storage like the rest of the system.
You could still do map-reduce operations, but for it to really fly, what you'd want is a side channel between the memory chips that allows the reduce to happen out of band from the front-side bus and the reduction to be sent to the CPU. And any workflow where you can stream the reduction to the CPU that would be even better for latency.