Earlier quoted context omitted.
You can build an x86 machine that can fully run DeepSeek R1 with 512GB VRAM for ~$2,500?
You will have to explain to me how.
Apple M3 Ultra
431–440 of 1001 posts
Re: Apple M3 Ultra
#432512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…
Why does it matter if you can run the LLM locally, if you're still running it on someone else's locked down computing platform?
Running locally, your data is not sent outside of your security perimeter off to a remote data center.
If you are going to argue that the OS or even below that the hardware could be compromised to still enable exfiltration, that is true, but it is a whole different ballgame from using an external SaaS no matter what the service guarantees.
Re: Apple M3 Ultra
#433Previous model of M2 Ultra had max memory of 192GB. Or 128GB for Pro and some other M3 model, which I think is plenty for even 99.9% of professional task. They now bump it to 512GB . Along with insane price tag of $9499 for 512GB Mac Studio. I am pretty sure this is some AI Gold rush.
Big question is: Does the $10k price already reflect Trump's tariffs on China? Or will the price rise further still..
Re: Apple M3 Ultra
#434Earlier quoted context omitted.
You can use Thunderbolt 5 interconnect (80Gbps) to run LLMs distributed across 4 or 5 Mac Studios.
But 80Gbit/s is way slower than even regular dual channel RAM, or am I missing something here? That would mean the LLM would be excruciatingly slow. You could get an old EPYC for a fraction of that price and have more performance.
The weights don't go over the network so performance is OK.
Re: Apple M3 Ultra
#435They update the Studio to M3 Ultra now, so M4 Ultra can presumably go directly into the Mac Pro at WWDC? Interesting timing. Maybe they'll change the form factor of the Mac Pro, too? Additionally, I would assume this is a very low-volume product, so it being on N3B isn't a dealbreaker. At the same time, these chips must be very expensive to make, so tying them with luxury-priced RAM makes some kind of sense.
Interestingly, Apple apparently confirmed to a French website that M4 lacks the interconnect required to make an "Ultra" [0][1], so contrary to what I originally thought, they maybe won't make this after all? I'll take this report with a grain of salt, but apparently it's coming directly from Apple.
Makes it even more puzzling what they are doing with the M2 Mac Pro.
[0] https://www.numerama.com/tech/1919213-m4-max-et-m3-ultra-let...
[1] More context on Macrumors: https://www.macrumors.com/2025/03/05/apple-confirms-m4-max-l...
Re: Apple M3 Ultra
#436512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…
Nvidia has had the Grace Hoppers for a while now. Is this not like that?
Re: Apple M3 Ultra
#437Earlier quoted context omitted.
I feel you on a lot of this! But out of the box Python support? Does anybody actually want that? It’s pretty darn quick & straightforward to get a Python environment up & running on MacOS. Maybe I’m misunderstanding what you mean here.
No one would want OOTB Python support. You'd be stuck on a version you didn't want to use.
I want it. That way, like code I write in any other language, it’ll run reliably on other people’s machines a few years from now.
I avoid writing python, so I’m usually the “other people” in that sentence.
Re: Apple M3 Ultra
#438We've come a long way since beowulf clusters of smart toasters.
Re: Apple M3 Ultra
#439512GB unified memory is absolutely wild for AI stuff! Compared to how many NVIDIA GPUs you would need, the pricing looks almost reasonable.
If you're going to overthrow your entire AI workflow to use a different API anyway, surely the AMD Instinct accelerator cards make more sense. They're expensive, but also a lot faster, and you don't need to deal with making your code work on macOS.
Doesn't AMD Instinct cost >$50K for 512GB?