Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Samsung's Processing-in-Memory (PIM)

chipsandcheese.com

Post reply on HN
Print view View original

101–110 of 120 posts

Re: Samsung's Processing-in-Memory (PIM)

#101

Post by zeusk »

therealcamino wrote: post #85
zeusk wrote: post #80

Earlier quoted context omitted.

why would it? the parent OS can already handle physically contiguous allocations so these should be no different (with the exception that a separate interface can be used to do compute over these buffers/pages).

If it requires physically contiguous RAM to work, then it's not really participating in the full virtual memory system, really. It would be using an exception to it, that can be accommodated to some extent by the OS, but not sitting in demand-paged storage like the rest of the system.

That really doesn't make sense because your system is already using memory that needs to be physically contiguous but mapped virtually. Also, would you say pinned memory is not part of the virtual memory system??

You're only restricted by the fragmentation of the system memory which is an issue yes, but it's dealt with in other ways.

Re: Samsung's Processing-in-Memory (PIM)

#102

Post by londons_explore »

PessimalDecimal wrote: post #64
danmaz74 wrote: post #35

Earlier quoted context omitted.

Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.

Turning RAM into an active coprocessor does feel like an awkward model for that though. Intel's Advanced Matrix Extensions (AMX) and the dedicated tile registers, which are basically systolic arrays IIUC, seem like a better fit. But Samsung doesn't have that option so they're playing the card they have.

I think useful flops per dollar will outweigh any "awkwardness" factor.

Elegant designs don't win anymore.

Re: Samsung's Processing-in-Memory (PIM)

#103

Post by zeusk »

teiferer wrote: post #94
therealcamino wrote: post #85

Earlier quoted context omitted.

If it requires physically contiguous RAM to work, then it's not really participating in the full virtual memory system, really. It would be using an exception to it, that can be accommodated to some extent by the OS, but not sitting in demand-paged storage like the rest of the system.

And the security folks will have tons of fun with this enormously leaky abstraction too.

Hey, just don't store your pagetables in PIM capable sections :)

Re: Samsung's Processing-in-Memory (PIM)

#104

Post by ActorNightly »

londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

Not if you have duplicates of rows on the first matrix, which can be done very efficiently if you build specialized hardware. Then its all just forward in parallel.

Re: Samsung's Processing-in-Memory (PIM)

#106

Post by touisteur »

po1nt wrote: post #62
bob1029 wrote: post #56

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…

I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module. Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region. Let's say you would need to run a sum over megabytes of data like for accounting. You…

Data movement and local operations are still bottlenecked today on memory bandwidth. Butterfly primitives, sorting/fft/1D-convolution, the whole cub library, could be ported there and have great performance wins. But the pain of programming and maintaining code using this...

Re: Samsung's Processing-in-Memory (PIM)

#107

Post by reliabilityguy »

ACCount37 wrote: post #98
reliabilityguy wrote: post #84

Earlier quoted context omitted.

How exactly the map-reduce will happen though? Won’t you need to do it host-side, or make a lot of reads and writes? Also, doesn’t it mean that you forgo batching?

Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.

Rollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place.

How would you do map-reduce across multiple DIMMs w/o extra reads/writes?

PIM implies some sort of distributed compute, which can work for some cases, but I am not sure LLMs are one of them.

Re: Samsung's Processing-in-Memory (PIM)

#108

Post by reliabilityguy »

imtringued wrote: post #34
reliabilityguy wrote: post #3

Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a lot. That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.

The idea is this: You have an eight socket server with 96 memory slots, you add 96x PIM memories into the server (optimistic), load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip. 614 GB/s x 96 = 58,944 GB/s. Alternatively, the memory is used for embedded inference tasks. You can now upgrade from the limited single or two digit MB SRAM accelerators to reasonably fast sing…

> load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip.

Won’t you have a bunch of extra reads/writes via the CPU because these DIMMs won’t be able to compute matrix multiplications?

Re: Samsung's Processing-in-Memory (PIM)

#110

Post by whatshisface »

reliabilityguy wrote: post #107
ACCount37 wrote: post #98

Earlier quoted context omitted.

Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.

Rollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place. How would you do map-reduce across multiple DIMMs w/o extra reads/writes? PIM implies some sort of distributed compute, which can work for so…

Re-arrange x+(y+(z+w)) into (x+y)+(z+w) to accomplish a distributed sum. Shared bus, interconnect fabric. This is a supercomputer on a chip.
Post reply on HN