Samsung's Processing-in-Memory (PIM) • Hacker News Board

Re: Samsung's Processing-in-Memory (PIM)

#41

Post by trentor » Sat, Aug 29, 2026, 11:04 AM UTC

stinkbeetle wrote: post #37

londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

> Whilst processing in memory is clearly the future How clear is that? The idea has been around for about 60 years, and many attempts made by people who thought the same thing. Maybe this time it'll be the future.

Because now the usecase fits the needs.

Re: Samsung's Processing-in-Memory (PIM)

#42

Post by hham » Sat, Aug 29, 2026, 11:07 AM UTC

memory plus memory bandwidth > GPU, this is the gist of it.

Re: Samsung's Processing-in-Memory (PIM)

#43

Post by samuelknight » Sat, Aug 29, 2026, 11:15 AM UTC

I saw them present a similar concept at Hot Chips in 2020 or 2021. It's still a cool idea, however people should remember that there are like 20 of these exotic accelerators designs pitched at trade shows every year that go nowhere.

Re: Samsung's Processing-in-Memory (PIM)

#44

Post by UltraSane » Sat, Aug 29, 2026, 11:15 AM UTC

harshaw wrote: post #39

This is somewhat orthogonal to the article, but the whole bubble on AI data centers seems to presume that the need for compute is so massive that it far exceeds the expected optimizations we would expect with at scale inference (PIM, ASICs, etc). I would expect that there is a set of optimizations like this one (or variations) that would someone negate the buildout. But it's not really discussed.

Jevons Paradox shows that increasing efficiency can increase demand for a product by making it cost effective for more uses.

Re: Samsung's Processing-in-Memory (PIM)

#45

Post by roryirvine » Sat, Aug 29, 2026, 11:18 AM UTC

Tenoke wrote: post #40

harshaw wrote: post #39

This is somewhat orthogonal to the article, but the whole bubble on AI data centers seems to presume that the need for compute is so massive that it far exceeds the expected optimizations we would expect with at scale inference (PIM, ASICs, etc). I would expect that there is a set of optimizations like this one (or variations) that would someone negate the buildout. But it's not really discussed.

There's been a ton of optimizations already, it hasn't remotely reduced demand even temporarily. More efficiency just makes the compute have even higher ROI per $ and watt spent.

With sufficient optimisation, there ought to be a tipping point beyond which local inference is good enough. And, sure, datacentre compute will still be needed for training but one of the biggest current uses will begin to taper off.

The question really is how soon we reach that tipping point, and whether it's before or after the current bubble runs out of steam for some other reason.

Re: Samsung's Processing-in-Memory (PIM)

#46

Post by eru » Sat, Aug 29, 2026, 11:24 AM UTC

amelius wrote: post #15

xyzzy123 wrote: post #5

Earlier quoted context omitted.

As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required. Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.

You: "AI, please write me $COOL_APP." AI: "Sorry, all the hardware is made for running AI."

We can run Doom on everything. Surely we can run some interesting apps on hardware that's originally made for AI. (One big moment for AI was when people figured out how to run it on hardware originally meant for Doom's successors.)

Re: Samsung's Processing-in-Memory (PIM)

#47

Post by dotancohen » Sat, Aug 29, 2026, 11:29 AM UTC

danmaz74 wrote: post #35

londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.

Exactly where I see this going as well. Sure, a smartphone might be a nice place to introduce such tech. But matrix multiplication is literally where all non-labour jobs are going - this is the bedrock for efficient (time, energy) machine learning and inference.

If AI really is going to eat all our jobs, then matrix multiplication in memory is almost a requirement.

Re: Samsung's Processing-in-Memory (PIM)

#48

Post by dotancohen » Sat, Aug 29, 2026, 11:30 AM UTC

OptionX wrote: post #31

So instead of putting more cache on the cpu you just put the cpu on the cache.

Sort of, but the cache architecture is replaced with memory architecture. But you could look at it either way.

Re: Samsung's Processing-in-Memory (PIM)

#49

Post by [deleted] » Sat, Aug 29, 2026, 11:32 AM UTC

UltraSane wrote: post #44

harshaw wrote: post #39

This is somewhat orthogonal to the article, but the whole bubble on AI data centers seems to presume that the need for compute is so massive that it far exceeds the expected optimizations we would expect with at scale inference (PIM, ASICs, etc). I would expect that there is a set of optimizations like this one (or variations) that would someone negate the buildout. But it's not really discussed.

Jevons Paradox shows that increasing efficiency can increase demand for a product by making it cost effective for more uses.

[deleted]

Re: Samsung's Processing-in-Memory (PIM)

#50

Post by hoppp » Sat, Aug 29, 2026, 11:47 AM UTC

imtringued wrote: post #34

reliabilityguy wrote: post #3

Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a lot. That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.

The idea is this: You have an eight socket server with 96 memory slots, you add 96x PIM memories into the server (optimistic), load all the LLM parameters or KV cache in RAM and exclusively let it perform GEMV and let it rip. 614 GB/s x 96 = 58,944 GB/s. Alternatively, the memory is used for embedded inference tasks. You can now upgrade from the limited single or two digit MB SRAM accelerators to reasonably fast sing…

And now the 96 memory slots need individual cooling.