Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Samsung's Processing-in-Memory (PIM)

chipsandcheese.com

Post reply on HN
Print view View original

61–70 of 118 posts

Re: Samsung's Processing-in-Memory (PIM)

#61

Post by trollbridge »

Eridrus wrote: post #58
bob1029 wrote: post #56

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…

In a world where AI is writing all of the code, the difficulty of the task may no longer be a blocker.

Indeed. SIMD type code used to be too hard for me to write very often, but now it’s easy to crank out tons of it.

Re: Samsung's Processing-in-Memory (PIM)

#62

Post by po1nt »

bob1029 wrote: post #56

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…

I think you could do many performant things without any involvement of software. For example you could do AVX on RAM. The CPU would recognize PID RAM and offload AVX instructions to the module.

Then, by simply asking for a special memory address you could have access to registers/regions within PID RAM that serve as a result region.

Let's say you would need to run a sum over megabytes of data like for accounting. You could just ask RAM to do it and load just the result. The bandwidth would could be 8x higher and software could stay the same.

Doing scalar operations, frequent dereferencing and similar would not get much peformance benefit in many cases, as loading and accessing CPU cache is often much faster. But simple vector operations over large data could be massive.

Having accelerators on RAM like for jpeg compression, audio decoding or mass data operations could be beneficial but you would need to be careful with heat dissipation.

Personally I'm a big fan of the "in-ram accelerator" idea especially for server space. Doing fuzzy search in RAM could be massive performance improvement.

Re: Samsung's Processing-in-Memory (PIM)

#64

Post by PessimalDecimal »

danmaz74 wrote: post #35
londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

Given how important matrix multiplication with a huge number of fixed parameters is becoming, there is an enormous incentive to design much more efficient architectures where this very simple compute is colocated with memory. Inference cost would come down a lot.

Turning RAM into an active coprocessor does feel like an awkward model for that though.

Intel's Advanced Matrix Extensions (AMX) and the dedicated tile registers, which are basically systolic arrays IIUC, seem like a better fit.

But Samsung doesn't have that option so they're playing the card they have.

Re: Samsung's Processing-in-Memory (PIM)

#65

Post by rivetfasten »

zozbot234 wrote: post #32
londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

"Movement is the main thing" is precisely why pursuing compute-in-RAM makes some sort of sense to begin with. But DRAM fabrication processes are quite specialized and do not perform well with pure compute logic. The overall profile of this thing will arguably be similar to a rather weak NPU, though with much better memory bandwidth - one key limitation, as with NPUs, will be the bespoke programming model and lack of…

I was under the impression they're doing a chiplet thing to get heterogenous processes for that reason.

Re: Samsung's Processing-in-Memory (PIM)

#67

Post by p1esk »

samuelknight wrote: post #43

I saw them present a similar concept at Hot Chips in 2020 or 2021. It's still a cool idea, however people should remember that there are like 20 of these exotic accelerators designs pitched at trade shows every year that go nowhere.

that go nowhere

But some do end up in the industry: Mythic AI, Encharge AI, d-Matrix.

Re: Samsung's Processing-in-Memory (PIM)

#68

Post by petra »

harshaw wrote: post #39

This is somewhat orthogonal to the article, but the whole bubble on AI data centers seems to presume that the need for compute is so massive that it far exceeds the expected optimizations we would expect with at scale inference (PIM, ASICs, etc). I would expect that there is a set of optimizations like this one (or variations) that would someone negate the buildout. But it's not really discussed.

It's not just inference, some things done in data centers like simulations, testing, are complementary to inference.

And in these types of hardware, the time between a successful prototype and a fully deployed product is pretty long. Maybe they count on that to know when to stop?

Re: Samsung's Processing-in-Memory (PIM)

#69

Post by HarHarVeryFunny »

I remember taking VLSI design as part of my Comp. Sci. degree at Bristol, UK c.1980, using the Conway & Mead book, and "Commingling of Processing and Memory" was mentioned even back then.

Obviously you (eventually) need your data where the compute is, especially in a non-von-Neumann architecture where moving data around isn't an option even if you were OK with the performance drop.

It seems kinda obvious that eventually AI will be implemented as low power dataflow custom chips integrating memory/state & compute, but who knows!

Re: Samsung's Processing-in-Memory (PIM)

#70

Post by jandrewrogers »

londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

People have been calling processing-in-memory "the future" since at least the 1980s. No one has been able to reduce the concept to a useful implementation but there is a long history of failed attempts.

At this point processing-in-memory has taken on the aura of fusion power.

Post reply on HN