What I find amusing about moving compute to a RAM bank is it _almost_ resembles where we were with ISA-based extended RAM back in the 1980's. Some cards featured a CPU that took over the whole system and/or functioned like an upgrade. Others were a "computer on a card" that provided other features. I think this goes to show how cyclic tech can be. So, something like Samsung's invention here might have gained traction, as overcoming the slow PC ISA bus would have been a huge accelerator, kind of like where we are now.
Samsung's Processing-in-Memory (PIM)
21–30 of 119 posts
Re: Samsung's Processing-in-Memory (PIM)
#22I wrote up a theoretical post here about LMM performance of a MacBook Pro with PIM memory: https://ben3d.ca/blog/m5-max-samsung-lpddr5-pim-650-tokens-p...
Interesting read. Definitely hoping this works out so we can have cheap LLM machines at home
Re: Samsung's Processing-in-Memory (PIM)
#23In-memory computation was already possible with regular DRAM: https://news.ycombinator.com/item?id=22712811 Add a new set of CPU instructions like “rep macb” ...and it's been long enough now, that I can say there was an effort to implement this on standard x86 memory controllers and have the existing string instructions do so, back in the days of SDR SDRAM, but the tradeoffs weren't (yet) in favour.
that's not at all comparable, you're still paying memory latency and not getting any extra bandwidth
Re: Samsung's Processing-in-Memory (PIM)
#24“ Each PIM block only has fast access to its locally attached DRAM bank. All other input data has to be brought in through the DRAM chip’s comparatively constrained external interface. PIM blocks can’t directly exchange data with each other, so the host has to move data using regular DRAM reads and writes if one PIM block needs to use results generated by another.”
So how big are these banks? If you can’t fit the weights of a layer into one bank then presumably you lose a lot of the speed gains.
Re: Samsung's Processing-in-Memory (PIM)
#25Whilst processing in memory is clearly the future, I am unconvinced by this implementation.
Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2).
To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a 'around the chip' ring shift register to pass every element of one matrix past every element of the other.
Re: Samsung's Processing-in-Memory (PIM)
#26“ Each PIM block only has fast access to its locally attached DRAM bank. All other input data has to be brought in through the DRAM chip’s comparatively constrained external interface. PIM blocks can’t directly exchange data with each other, so the host has to move data using regular DRAM reads and writes if one PIM block needs to use results generated by another.” So how big are these banks? If you can’t fit the wei…
Not necessarily. If your weights have to go across 2 banks, you just have to split and transfer the input and output vectors, which are much smaller.
Re: Samsung's Processing-in-Memory (PIM)
#27Feels like the most realistic/short-term way to make use of this would be to set up some barebones RTOS to run from CPU cache with the PIM memory being used for compute only and use the device as a network attached accelerator.
Re: Samsung's Processing-in-Memory (PIM)
#28“ Each PIM block only has fast access to its locally attached DRAM bank. All other input data has to be brought in through the DRAM chip’s comparatively constrained external interface. PIM blocks can’t directly exchange data with each other, so the host has to move data using regular DRAM reads and writes if one PIM block needs to use results generated by another.” So how big are these banks? If you can’t fit the wei…
For GEMV you lose nothing.
The reason is quite simple. You can split the matrix along both dimensions so you just tile it into 64x64 or whatever fits into the bank and just fill it up. The biggest problem is load balancing the tiles across all banks for maximum parallelism.
For GEMM I believe there is no point in doing PIM, you are better off with a GPU or NPU.
Re: Samsung's Processing-in-Memory (PIM)
#29What about energy consumption? Wouldn't active cooling be needed for RAM as well as for CPU and GPU?
Re: Samsung's Processing-in-Memory (PIM)
#30Self changing RAM and a complex way to interact with it in software. A new security nightmare is emerging.