Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
1–10 of 62 posts
Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
#2Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
#3Let's also ensure the SSD doesn't age prematurely.
Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
#4Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
Yeah this is such a funny thing going around. Try to run or train a small/medium sized model on a MacBook. It doesn’t go very good compared to a dedicated gpu
This is likely the right path in the future but it isn’t there yet today
Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
#5Can't really run it as well, though. My "mini PC" is an M4 Max with 128GB of unified memory and the memory bandwidth is still sorely lacking for inference (although it's far better than any non-unified consumer architecture!).
To be fair it's "only" half the throughput of a 4090 and a third of an RTX 6000. Significant but not an order of magnitude.
Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
#6Let's also ensure the SSD doesn't age prematurely.
I was under the impression that when you're streaming the weights from disk because the full model won't fit in memory, that it is solely reading from the SSD, not writing, so it wouldn't be causing wear on your SSD.
Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
#7Think future generations of AMD could get quite interesting. They’re no doubt seeing people whining about mem throughput specifically
Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
#8"Can't" is not really correct.
Nowadays, specially with MoE models you can run parts of the model on GPU and still get some speed up.
Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
#9[flagged]
Re: Unified Memory, Explained: Why Mini PCs Can Run 70B Models a Big GPU Can't
#10[flagged]
Maybe like… focus on the actual content instead of perceived writing patterns? Crazy I know.