Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Samsung's Processing-in-Memory (PIM)

chipsandcheese.com

Post reply on HN
Print view View original

111–120 of 120 posts

Re: Samsung's Processing-in-Memory (PIM)

#111

Post by zerd »

dannyw wrote: post #53
petu wrote: post #17

Earlier quoted context omitted.

Yes, but running out of RAM is impractical due to low memory bandwidth. According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width: > Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8…

You can scale with more memory channels. Workstation/server platforms go up to 12 or 16 channels if I remember correctly. Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.

How do you scale past 16 channels though? 16 channels give you around 614 GB/s, while PIM can do that per chip, so it can achieve 58TB/s.

Re: Samsung's Processing-in-Memory (PIM)

#112

Post by rapsey »

jandrewrogers wrote: post #70
londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

People have been calling processing-in-memory "the future" since at least the 1980s. No one has been able to reduce the concept to a useful implementation but there is a long history of failed attempts. At this point processing-in-memory has taken on the aura of fusion power.

OLED also has a deep history with a pretty famous opinions that it is impossible and a waste of time and money for companies to invest in the development. So does AI. From inception to product it is normal for decades to pass.

Compute needs change. AI needs are pretty unique in terms of scale and type of compute requirements to anything else so far.

Re: Samsung's Processing-in-Memory (PIM)

#113

Post by fulafel »

tesnorindian wrote: post #54

In memory compute has been in talks since LLMs took up. I remember few flocks were trying to get RISC V cores in the memory like these papers https://arxiv.org/abs/2602.01827

It has been a prominent "next major shift" idea in computer architecture since the 90s, to deal with the memory wall. Eg David Patterson advocating it in the 1990 and 1997 articles. or CRAM [1].

[1] https://www.eecg.toronto.edu/~stumm/Theses/Elliott-PhD98.pdf

Re: Samsung's Processing-in-Memory (PIM)

#114

Post by skissane »

consp wrote: post #7

So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).

> So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).

This is a temporary issue. JEDEC's LPDDR6-PIM is going to add defined commands for Processing-in-Memory operations. Once there are standardised commands, it will be possible for the CPU vendors to make the CPU cache aware of what is happening.

Of course, that doesn't solve it for this generation of the technology. But I think this generation is more of a demo for early adopters to gain experience with it. It will likely take a few years for all these issues to be solved, but there is no principled reason why they can't be.

Re: Samsung's Processing-in-Memory (PIM)

#115

Post by shuwix »

jandrewrogers wrote: post #70
londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

People have been calling processing-in-memory "the future" since at least the 1980s. No one has been able to reduce the concept to a useful implementation but there is a long history of failed attempts. At this point processing-in-memory has taken on the aura of fusion power.

I think Wright brothers had long history of failed attempts.

And all of sudden, no passengers are crossing Atlantic on ship.

Well, when you fail at everything, you want other's to fail too. Luckily for us, your failure doesn't affect others.

Re: Samsung's Processing-in-Memory (PIM)

#116

Post by skissane »

bob1029 wrote: post #56

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…

One thing this makes possible – accelerated memcpy. Of course, it is only worthwhile once the memory buffer to be copied is large enough. But the copy speed could be greatly increased if it could all be done inside the memory module. And of course, that only works if the source and destination buffers are physically on the same module. But, if your application allocates 1GB pages, the OS could attempt to ensure they are on the same memory module.

Accelerated memcpy is already offered by some Intel server systems (QuickData aka IOATDMA aka CBDMA aka SDMA), but it uses a memory-to-memory DMA engine on the CPU die, so the memory to be copied still has to travel back and forward between the CPU and the memory modules via the memory controller, even though using a DMA engine means it isn't consuming a CPU core. With this, the same process could be made much faster, bypassing the CPU entirely, provided the source and destination are on the same module.

Re: Samsung's Processing-in-Memory (PIM)

#117

Post by stinkbeetle »

trentor wrote: post #41
stinkbeetle wrote: post #37

Earlier quoted context omitted.

> Whilst processing in memory is clearly the future How clear is that? The idea has been around for about 60 years, and many attempts made by people who thought the same thing. Maybe this time it'll be the future.

Because now the usecase fits the needs.

This is really just restating the assertion I'm asking about. Previous work also thought their use cases fit the needs.

Everything changes all the time in computing, so there are lots of differences. I'm not asking if things are different or claiming it won't work this time either. I'm asking what it is this time that makes it different / obvious. Not rhetorical, I'm interested if someone can actually explain. Preferably with trends and numbers that have $ and flops and picojoules and gates and bits in their units.

Re: Samsung's Processing-in-Memory (PIM)

#118

Post by ACCount37 »

reliabilityguy wrote: post #107
ACCount37 wrote: post #98

Earlier quoted context omitted.

Map-reduce is implemented as a rolling calc, see: online softmax in FlashAttention kernels.

Rollie calculation, like the online softmax in FA, implies a centralized computing unit that does the compute and stores the intermediate results in its registers. With PIM you have no centralized compute unit, you have a bunch of memory, and a bunch of MACs all over the place. How would you do map-reduce across multiple DIMMs w/o extra reads/writes? PIM implies some sort of distributed compute, which can work for so…

Is there a single reason why we can't just "distribute" the online softmax?

Each die-attached PIM accelerator computes online softmax for its own KVs. Then the central unit gathers the softmax intermediates, one intermediate per die, and uses those to compute the final softmax.

The PIM win is that we crater the memory traffic between the central accelerator and the memory dies for attention ops. Most of the attention bandwidth never leaves the memory.

This isn't "run the entire LLM in PIM", no - this is "offload the parts of LLM that benefit from PIM the most to PIM".

Re: Samsung's Processing-in-Memory (PIM)

#119

Post by BiraIgnacio »

HarHarVeryFunny wrote: post #69

I remember taking VLSI design as part of my Comp. Sci. degree at Bristol, UK c.1980, using the Conway & Mead book, and "Commingling of Processing and Memory" was mentioned even back then. Obviously you (eventually) need your data where the compute is, especially in a non-von-Neumann architecture where moving data around isn't an option even if you were OK with the performance drop. It seems kinda obvious that eventua…

Very cool paradigm, I wonder if it's easier to have processing moved/added to memory or have more memory in the CPU.

In either case, as you said, it would be having both data and compute closer than they are right now.

Post reply on HN