Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Samsung's Processing-in-Memory (PIM)

chipsandcheese.com

Post reply on HN
Print view View original

51–60 of 118 posts

Re: Samsung's Processing-in-Memory (PIM)

#51

Post by Tenoke »

roryirvine wrote: post #45
Tenoke wrote: post #40

Earlier quoted context omitted.

There's been a ton of optimizations already, it hasn't remotely reduced demand even temporarily. More efficiency just makes the compute have even higher ROI per $ and watt spent.

With sufficient optimisation, there ought to be a tipping point beyond which local inference is good enough. And, sure, datacentre compute will still be needed for training but one of the biggest current uses will begin to taper off. The question really is how soon we reach that tipping point, and whether it's before or after the current bubble runs out of steam for some other reason.

>there ought to be a tipping point beyond which local inference is good enough

There's no such ought really. Even at current levels you'd need like a 100x gain from here to approach current top proprietary models (probably a lot more for say Mythos or Mythos 2), and it's not like they are stoppng to improve. This is before we even account that you'd just be running 1 agent then, and not a swarm like you'd be able to in the cloud or that you can do only so much compression before you are losing out

Re: Samsung's Processing-in-Memory (PIM)

#52

Post by topspin »

plywoodShadow wrote: post #29

What about energy consumption? Wouldn't active cooling be needed for RAM as well as for CPU and GPU?

The better question is: what is the net gain for the overall system? If PIM reduces the net thermal load and power consumption of the system for the same workload, then it’s a win regardless of where the heat sinks end up. The customers Samsung has in mind for this today are not limited to commodity designs. They’re using novel designs with each new hardware generation, so moving heat sinks around is not a deal-breaker.

Re: Samsung's Processing-in-Memory (PIM)

#53

Post by dannyw »

petu wrote: post #17
embedding-shape wrote: post #9

Earlier quoted context omitted.

> Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits. Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.

Yes, but running out of RAM is impractical due to low memory bandwidth. According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width: > Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8…

You can scale with more memory channels. Workstation/server platforms go up to 12 or 16 channels if I remember correctly.

Consumer platforms have been stuck at dual channel for decades; most of it I attribute to intentional product segmentation. I'm hoping that LLMs might change eventually for an upcoming consumer platforms; going to 4 channel would be really nice.

Re: Samsung's Processing-in-Memory (PIM)

#55

Post by [deleted] »

londons_explore wrote: post #26
mr_toad wrote: post #24

“ Each PIM block only has fast access to its locally attached DRAM bank. All other input data has to be brought in through the DRAM chip’s comparatively constrained external interface. PIM blocks can’t directly exchange data with each other, so the host has to move data using regular DRAM reads and writes if one PIM block needs to use results generated by another.” So how big are these banks? If you can’t fit the wei…

Not necessarily. If your weights have to go across 2 banks, you just have to split and transfer the input and output vectors, which are much smaller.

[deleted]

Re: Samsung's Processing-in-Memory (PIM)

#56

Post by bob1029 »

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applications noted above eventually got their own flavors.

I think the Von Neumann bottleneck is mostly a feature. The fact that communication of information across distances is expensive should not be immediately assumed to mean that it is universally flawed to do this. You are paying for something when you use all those joules. I'd argue we are usually wasting our energy with regard to information communication (e.g., lighting up a network interface & copper because we couldn't be bothered to use SQLite), but other times this stuff is fundamentally required for practical solutions to exist.

Re: Samsung's Processing-in-Memory (PIM)

#57

Post by [deleted] »

londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

[deleted]

Re: Samsung's Processing-in-Memory (PIM)

#58

Post by Eridrus »

bob1029 wrote: post #56

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…

In a world where AI is writing all of the code, the difficulty of the task may no longer be a blocker.

Re: Samsung's Processing-in-Memory (PIM)

#59

Post by convolvatron »

londons_explore wrote: post #25

Whilst processing in memory is clearly the future, I am unconvinced by this implementation. Matrix multiplication involves getting every entry of the input and output matrices to be at the same multiplier at the same time. (Ie. N^2). To do that, a lot of data movement needs to happen. Movement is the main thing - the multiplication and addition is a sideshow as far as energy and silicon space is concerned. You need a…

you're absolutely correct that pim without a real discussion about how that works in a broader communications context is kind of useless.

what I find strange is the adoption of a standard synchronous dram interface. that's a horrible left over piece of architecture that severely constrains the applicability of this device. control flow on the dram side can't initiate any transactions on its own, or respond after work has been done - its like usb, except with a hard limit on the response.

that severely limits the utility of the in-memory processors to doing things like encryption and compression - but even then those impose delays that effect the consistency model across that interface.

Re: Samsung's Processing-in-Memory (PIM)

#60

Post by jeffbee »

bob1029 wrote: post #56

The tradeoff with putting the compute in the memory is that you have to know exactly where the dependent information will be at all times. Most problems do not fit this pattern very well. AI, gaming and crypto being the most obvious exceptions. It is incredibly constraining to develop applications using specialized hardware like this. You might as well spin out an ASIC for whatever it is you are doing. All 3 applicat…

There are fundamental issues here and I think the article only touched on a few. On the software side this completely blows up the whole virtual memory concept. We will need different operating systems.
Post reply on HN