If we could buy a 64gb stick and run a 32b model with 30tps on it. This could sell
Samsung's Processing-in-Memory (PIM)
11–20 of 120 posts
Re: Samsung's Processing-in-Memory (PIM)
#12So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).
I guess you also get very high bandwidth that way? I‘m not sure that would come for free though.
Re: Samsung's Processing-in-Memory (PIM)
#13[dead]
Re: Samsung's Processing-in-Memory (PIM)
#14So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).
if your working set fits in cache PIM is irrelevant
Re: Samsung's Processing-in-Memory (PIM)
#15Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a lot. That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.
As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required. Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
You: "AI, please write me $COOL_APP."
AI: "Sorry, all the hardware is made for running AI."
Re: Samsung's Processing-in-Memory (PIM)
#16So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).
if your working set fits in cache PIM is irrelevant
Caches are important for cpu core performance even when the the working set doesn't fit.
Re: Samsung's Processing-in-Memory (PIM)
#17Earlier quoted context omitted.
As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required. Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
> Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits. Isn't this how it works today already? Granted you wanted to run it on RAM rather than VRAM.
Yes, but running out of RAM is impractical due to low memory bandwidth.
According to the article/Samsung RAM dies inside can support way higher bandwidth than they expose, they're limited by external interface / bus width:
> Together, they can utilize the chip’s internal bandwidth across all 16 banks, which comes out to 614 GB/s. For comparison, regular DRAM accesses can hit two banks in parallel and max out at 76.8 GB/s.
And that's just for single 64-bit IC. So way faster and more power efficient.
Re: Samsung's Processing-in-Memory (PIM)
#18So you basically dispose of cache for the memory region used? I wonder what the offsets of the cache misses is going to be in practice (the article addresses it but there is no solution/impact given by samsung).
I guess you also get very high bandwidth that way? I‘m not sure that would come for free though.
It comes at a cost of a fragmented memory space, which is fine for some applications, like LLMs
Re: Samsung's Processing-in-Memory (PIM)
#19Interesting that Samsung still pursues PIM. IIRC they had a paper in ISCA21 or 22 where they showed HBM2 module with PIM, which back then impressed me quite a lot. That being said, I am not sure what’s the killer application for this technology, and without such application adoption is unlikely.
As I understand it, the killer app is llms. You could run MACs directly in RAM, offloading a lot of work from CPU and cutting down on insane (external) memory bandwidth required. Imagine (this is a fantasy pitch but potentially achievable for some use cases) wanting to run a larger llm and all you have to do is buy more RAM so it fits.
[dead]
Re: Samsung's Processing-in-Memory (PIM)
#20I wrote up a theoretical post here about LMM performance of a MacBook Pro with PIM memory:
https://ben3d.ca/blog/m5-max-samsung-lpddr5-pim-650-tokens-p...