Apple M3 Ultra • Hacker News Board

Re: Apple M3 Ultra

#351

Post by cynicalpeace » Wed, Mar 05, 2025, 4:13 PM UTC

bigyabai wrote: post #320

cynicalpeace wrote: post #300

Can someone explain what it would take for Apple to overtake NVIDIA as the preferred solution for AI shops? This is my understanding (probably incorrect in some places) 1. NVIDIA's big advantage is that they design the hardware (chips) and software (CUDA). But Apple also designs the hardware (chips) and software (Metal and MacOS). 2. CUDA has native support by AI libraries like PyTorch and Tensorflow, so works extra…

It's still boiling down to hardware and software differences. In terms of hardware - Apple designs their GPUs for GPU workloads, whereas Nvidia has a decades-old lead on optimizing for general-purpose compute. They've gotten really good at pipelining and keeping their raster performance competitive while also accelerating AI and ML. Meanwhile, Apple is directing most of their performance to just the raster stuff. The…

> keeping their raster performance competitive while also accelerating AI and ML. Meanwhile, Apple is directing most of their performance to just the raster stuff. They could pivot to an Nvidia-style design, but that would be pretty unprecedented (even if a seemingly correct decision).

Isn't Apple also focusing on the AI stuff? How has it not already made that decision? What would prevent Apple from making that decision?

> Metal does boast some compute features, but you're expected to write most of the support yourself in the form of compute shaders. This is a pretty radical departure from the pre-rolled, almost "cargo cult" CUDA mentality.

Can you give an example of where Metal wants you to write something yourself whereas CUDA is pre-rolled?

Re: Apple M3 Ultra

#352

Post by buildbot » Wed, Mar 05, 2025, 4:13 PM UTC

jeffbee wrote: post #118

InTheArena wrote: post #97

Earlier quoted context omitted.

That's CPU only memory, not high bandwidth, and not addressable by the GPU.

There isn't anything particularly high-bandwidth about Apple's DDR5 implementation, either. They just have a lot of channels, which is why I compared it to a 24-channel EPYC system. I agree that their integrated GPU architecture hits a unique design point that you don't get from nvidia, who prefer to ship smaller amounts of very different kinds of memory. Apple's architecture may be more suited to some workloads but…

M3 Ultra has 819GB/s, and a single epyc cpu with 12 channels has 460GB/s. As far as I know, llama.cpp and friends don’t scale across multiple sockets so you can’t use a dual socket Turin system to match the M3 Ultra.

Also, 32GB DDR5 RDIMMS are ~200, so that’s 5K for 24 right there. Then you need 2x CPUs, at ~1K for the cheapest, and you need 2, and then a motherboard that’s another 1K. So for 8K (more, given you need a case, power supply, and cooling!), you get a system with about half the memory bandwidth, much higher power consumption, and very large.

Re: Apple M3 Ultra

#353

Post by 827a » Wed, Mar 05, 2025, 4:13 PM UTC

DrBenCarson wrote: post #289

827a wrote: post #164

Earlier quoted context omitted.

Is this actually true? Were people doing this with the 192gb of the M2 Ultra? I'm curious to learn how AI shops are actually doing model development if anyone has experience there. What I imagined was: Its all in the "cloud" (or, their own infra), and the local machine doesn't matter. If it did matter, the nvidia software stack is too important, especially given that a 512gb M3 Ultra config costs $10,000+.

You’re largely correct for training models Where this hardware shines is inference (aka developing products on top of the models themselves)

True. But with Project Digits supposedly around the corner, which supposedly costs $3,000 and supports ConnectX and runs Blackwell; what's the over-under on just buying two of those at about half the price of one maxed M3 Ultra Mac Studio?

Re: Apple M3 Ultra

#354

Post by whimsicalism » Wed, Mar 05, 2025, 4:13 PM UTC

cubefox wrote: post #348

whimsicalism wrote: post #278

Earlier quoted context omitted.

Transformers are typically memory- bandwidth bound during decoding. This chip is going to have a much worse memory b/w than the nvidia chips. My guess is that these chips could be compute-bound though given how little compute capacity they have.

> Transformers are typically memory-bandwidth bound during decoding. Not in case of language models, which are typically bound by memory size rather than bandwidth.

nope

Re: Apple M3 Ultra

#355

Post by atwrk » Wed, Mar 05, 2025, 4:14 PM UTC

spiderfarmer wrote: post #255

bigyabai wrote: post #233

Earlier quoted context omitted.

For enterprise markets, this is table stakes. A lot of datacenter customers will probably ignore this release altogether since there isn't a high-bandwidth option for systems interconnect.

You can use Thunderbolt 5 interconnect (80Gbps) to run LLMs distributed across 4 or 5 Mac Studios.

But 80Gbit/s is way slower than even regular dual channel RAM, or am I missing something here? That would mean the LLM would be excruciatingly slow. You could get an old EPYC for a fraction of that price and have more performance.

Re: Apple M3 Ultra

#356

Post by re-thc » Wed, Mar 05, 2025, 4:15 PM UTC

dylan604 wrote: post #204

re-thc wrote: post #166

Earlier quoted context omitted.

> So if someone wants to buy Apple hardware to run Linux, it does not have a negative affect to AAPL It does. Support costs. How do you prove it's a hardware failure or software? What should they do? Say it "unofficially" supports Linux? People would still try to get support. Eventually they'd have to test it themselves etc.

Apple has already been in this spot. With the TrashCan MacPro, there was an issue with DaVinci Resolve under OS X at the time where the GPU was cause render issues. If you then rebooted into Windows with BootCamp using the exact same hardware and open up the exact same Resolve project with the exact same footage, the render errors disappeared. Apple blamed Resolve. DaVinci blamed GPU drivers. GPU blamed Apple.

> Apple has already been in this spot.

Has been. This is importance. Past tense. Maybe that's the point - they gave up on it acknowledging the extra costs / issues.

Re: Apple M3 Ultra

#357

Post by johntitorjr » Wed, Mar 05, 2025, 4:16 PM UTC

NightlyDev wrote: post #178

johntitorjr wrote: post #40

Lots of AI HW is focused on RAM (512GB!). I have a cost-sensitive application that needs speed (300+ TOPS), but only 1GB of RAM. Are there any HW companies focused on that space?

Most recent GPUs will do. An older RTX 4070 is over 400 TOPS, the new RTX 5070 is around 1000 TOPS, and the RTX 5090 is around 3600 TOPS.

Yeah, that's basically where I'm at with options. Not ideal for a cost sensitive application.

Re: Apple M3 Ultra

#358

Post by [deleted] » Wed, Mar 05, 2025, 4:16 PM UTC

FloatArtifact wrote: post #301

cxie wrote: post #222

512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…

They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigabytes of uRAM for 10 grand. The the question is if a llm will run with usable performance at that scale? The point is there's diminishing returns despite having enough uRAM with the same amount of memory bandwidth even with increased processing speed of the new…

[deleted]

Re: Apple M3 Ultra

#359

Post by petecooper » Wed, Mar 05, 2025, 4:17 PM UTC

jeffhuys wrote: post #328

petecooper wrote: post #173

Earlier quoted context omitted.

>Transmission is the app. Former Transmission user here. I realise you didn't ask, but you might find some improvements in qBittorrent.

I went to Transmission years and years ago because it's just simple. It has all the options if you need them, but no HUUUGE interface with RSS feeds, 10001 stats about your download, categories, tags, etc etc etc. Transmission is just a small, floating window with your downloads. Click for more. It fits in the macOS vibe. But I'm a person that fully adopted the original macOS "way of working" - kicked the full-screen…

>why would you go FROM Transmission to qBittorrent?

In my case: some torrents wouldn't find known-good seeds in Transmission but worked fine in qBittorrent; there's reasonable (but not perfect) support for libtorrent 2.0 in qBittorrent; my download speeds and overall responsiveness is anecdotally better in qBittorrent, and; I make use of some of the nitty gritty settings in qBittorrent.

Re: Apple M3 Ultra

#360

Post by hedora » Wed, Mar 05, 2025, 4:17 PM UTC

PaulHoule wrote: post #326

exabrial wrote: post #203

Earlier quoted context omitted.

If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo. The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.

If I read this right, the r8g.48xlarge at AMZN [1] has 192 cores and 1536GB which exceeds the M3 Ultra in some metrics. It reminds me of the 1990s when my old school was using Sun machines based on the 68k series and later SPARC and we were blown away with the toaster-sized HP PA RISC machine that was used for student work for all the CS classes. Then Linux came out and it was clear the 386 trashed them all in terms…

In fairness, the sun and dec boxes I used back then (up to about 1999) could hold their own against intel machines.

Then, one day, we built a 5 machine amd athlon xp linux cluster for $2000 ($400/machine) that beat all the unix and windows server hardware by at least 10x on $/perf.

It’s nice that we have more than one viable cpu vendor these days, though it seems like there’s only one viable fab company.