Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Samsung's Processing-in-Memory (PIM)

chipsandcheese.com

Post reply on HN
Print view View original

71–80 of 119 posts

Re: Samsung's Processing-in-Memory (PIM)

#71

Post by vardump »

pragma_x wrote: post #21

What I find amusing about moving compute to a RAM bank is it _almost_ resembles where we were with ISA-based extended RAM back in the 1980's. Some cards featured a CPU that took over the whole system and/or functioned like an upgrade. Others were a "computer on a card" that provided other features. I think this goes to show how cyclic tech can be. So, something like Samsung's invention here might have gained traction…

Maybe we can connect multiple Samsung RAM sticks to a grid and call it RAMsputer.

Re: Samsung's Processing-in-Memory (PIM)

#72

Post by bob1029 »

po1nt wrote: post #62
bob1029 wrote: post #56

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…

I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module. Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region. Let's say you would need to run a sum over megabytes of data like for accounting. You…

> Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.

I think we are essentially reinventing SSE, AVX & friends from first principles. This is already being done. Compare the speed of libjpegturbo to a non-vectorized implementation and you'll find a 2-4x difference in throughput.

Re: Samsung's Processing-in-Memory (PIM)

#74

Post by throwaway173738 »

hoppp wrote: post #50
imtringued wrote: post #34

Earlier quoted context omitted.

The idea is this: You have an eight socket server with 96 memory slots, you add 96x PIM memories into the server (optimistic), load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip. 614 GB/s x 96 = 58,944 GB/s. Alternatively, the memory is used for embedded inference tasks. You can now upgrade from the limited single or two digit MB SRAM accelerators to reasonably fast sing…

And now the 96 memory slots need individual cooling.

Which might be easier since the surface is larger

Re: Samsung's Processing-in-Memory (PIM)

#75

Post by EPWN3D »

po1nt wrote: post #62
bob1029 wrote: post #56

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…

I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module. Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region. Let's say you would need to run a sum over megabytes of data like for accounting. You…

Yeah this sounds a lot like a natural evolution of SIMD for me, just cut out the middleman and put the SIMD units straight into RAM.

I can imagine some power savings for always-on display applications too. Rather than periodically waking the CPU/GPU to update the frame buffer, you can just stash small bits of periodic logic in memory (e.g. move the second hand of a clock).

Re: Samsung's Processing-in-Memory (PIM)

#76

Post by [deleted] »

netfortius wrote: post #6
reliabilityguy wrote: post #3

Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a lot. That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.

> That being said, I am not sure what's the killer application for this technology... Build it, and they will come ;)

[deleted]

Re: Samsung's Processing-in-Memory (PIM)

#77

Post by eureka7 »

Tenoke wrote: post #51
roryirvine wrote: post #45

Earlier quoted context omitted.

With sufficient optimisation, there ought to be a tipping point beyond which local inference is good enough. And, sure, datacentre compute will still be needed for training but one of the biggest current uses will begin to taper off. The question really is how soon we reach that tipping point, and whether it's before or after the current bubble runs out of steam for some other reason.

>there ought to be a tipping point beyond which local inference is good enough There's no such ought really. Even at current levels you'd need like a 100x gain from here to approach current top proprietary models (probably a lot more for say Mythos or Mythos 2), and it's not like they are stoppng to improve. This is before we even account that you'd just be running 1 agent then, and not a swarm like you'd be able to…

> (probably a lot more for say Mythos or Mythos 2)

Not everyone needs that large of a model, though.

Re: Samsung's Processing-in-Memory (PIM)

#78

Post by saltcured »

EPWN3D wrote: post #75
po1nt wrote: post #62

Earlier quoted context omitted.

I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module. Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region. Let's say you would need to run a sum over megabytes of data like for accounting. You…

Yeah this sounds a lot like a natural evolution of SIMD for me, just cut out the middleman and put the SIMD units straight into RAM. I can imagine some power savings for always-on display applications too. Rather than periodically waking the CPU/GPU to update the frame buffer, you can just stash small bits of periodic logic in memory (e.g. move the second hand of a clock).

I'm having a hard time imagining this SIMD replacement except for extremely narrow use cases. Are you suggesting the PIM would have a full blown IO controller and cache subsystem to fetch remote operands?

I assume PIM is only going to work well for chunky streaming over the data within that particular memory module. Something that address and operate on whole rows at once and has minimal buffering between the RAM access ports and the PIM register state.

A lot of SIMD code can be on two array operands, and I expect this PIM approach only works well if both are stored locally in the same memory "local" to the PIM and where it can efficiently interleave at the natural addresisng and access widths. Too much random access or needing "remote" data sounds like the point where PIM fails and you need the elaborate memory IO controllers and caching subsystems of CPUs sitting on top of the distributed memory modules..?

Re: Samsung's Processing-in-Memory (PIM)

#80

Post by zeusk »

jeffbee wrote: post #60
bob1029 wrote: post #56

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…

There are fundamental issues here and I think the article only touched on a few. On the software side this completely blows up the whole virtual memory concept. We will need different operating systems.

why would it? the parent OS can already handle physically contiguous allocations so these should be no different (with the exception that a separate interface can be used to do compute over these buffers/pages).
Post reply on HN