Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

551–560 of 1001 posts

Re: Apple M3 Ultra

#551

Post by rbanffy »

kllrnohj wrote: post #347
stego-tech wrote: post #248

Earlier quoted context omitted.

I cannot express how dirt cheap that pricepoint is for what's on offer, especially when you're comparing it to rackmount servers. By the time you've shoehorned in an nVidia GPU and all that RAM, you're easily looking at 5x that MSRP; sure, you get proper redundancy and extendable storage for that added cost, but now you also need redundant UPSes and have local storage to manage instead of centralized SANs or NASes. F…

> By the time you've shoehorned in an nVidia GPU and all that RAM, you're easily looking at 5x that MSRP That nvidia GPU setup will actually have the compute grunt to make use of the RAM, though, which this M3 Ultra probably realistically doesn't. After all, if the only thing that mattered was RAM then the 2TB you can shove into an Epyc or Xeon would already be dominating the AI industry. But they aren't, because it…

Had the M3 GPU been much wider, it would be constrained by the memory bandwidth. It might still have an advantage over Nvidia competitors in that it has 512GB accessible to it and will need to push less memory across socket boundaries.

It all depends on the workload you want to run.

Re: Apple M3 Ultra

#552

Post by amelius »

dylan604 wrote: post #117
c0deR3D wrote: post #82

When would Apple silicons made natively support for OSes such as Linux? Apple seemlingly reluctant to release detailed technical reference manual for M-series SoCs, which makes running Linux natively on Apple silicon challenging.

That’s what’s weird to me too. It’s not like they would lose sales of macOS as it is given away with the hardware. So if someone wants to buy Apple hardware to run Linux, it does not have a negative affect to AAPL

But then they'd have to open up their internal documentation of their silicon, which could possibly be a legal disaster (patents).

Re: Apple M3 Ultra

#554

Post by matt-p »

kllrnohj wrote: post #513
matt-p wrote: post #453

Earlier quoted context omitted.

Not really like for like. The pricing isn't as insane as you'd think, 96 to 256GB is 1500 which isn't 'cheap' but, it could be worse. All in 5,500 gets you a ultra with 256GB memory, 28 cores, 60 GPU cores, 10Gb network - I think you'd be hard pushed to build a server for less.

5,500 easily gets me either vastly more CPU cores if I care more about that or a vastly faster GPU if I care more about that. Or for both a 9950x + 5090 (assuming you can actually find one in stock) is ~$3000 for the pair + motherboard, leaving a solid $2500 for whatever amount of RAM, storage, and networking you desire. The M3 strikes a very particular middle ground for AI of lots of RAM but a significantly slower G…

That config would also use about 10x the power, and you still wouldn't be able to run a model over 32GB whereas the studio can easily cope with 70B llama and plenty of space to grow.

I think it actually is perfect for local inference in a way that build or any other pc build in this price range would be.

Re: Apple M3 Ultra

#555

Post by kllrnohj »

matt-p wrote: post #554
kllrnohj wrote: post #513

Earlier quoted context omitted.

5,500 easily gets me either vastly more CPU cores if I care more about that or a vastly faster GPU if I care more about that. Or for both a 9950x + 5090 (assuming you can actually find one in stock) is ~$3000 for the pair + motherboard, leaving a solid $2500 for whatever amount of RAM, storage, and networking you desire. The M3 strikes a very particular middle ground for AI of lots of RAM but a significantly slower G…

That config would also use about 10x the power, and you still wouldn't be able to run a model over 32GB whereas the studio can easily cope with 70B llama and plenty of space to grow. I think it actually is perfect for local inference in a way that build or any other pc build in this price range would be.

The M3 Ultra studio also wouldn't be able to run path traced Cyberpunk at all no matter how much RAM it has. Workloads other than local inference LLMs exist, you know :) After all, if the only thing this was built to do was run LLMs then they wouldn't have bothered adding so many CPU cores or video engines. CPU cores (along with networking) being 2 of the specs highlighted by the person I was responding to, so they were obviously valuing more than just LLM use cases.

Re: Apple M3 Ultra

#556

Post by lanceflt »

bearjaws wrote: post #327
behnamoh wrote: post #98

Earlier quoted context omitted.

> I can't wait to see someone testing the full DeepSeek model on this at 819 GB per second bandwidth, the experience would be terrible

Not sure why you are being downvoted, we already know the performance numbers due to memory bandwidth constraints on the M4 Max chips, it would apply here as well. 525GB/s to 1000GB/s will double the TPS at best, which is still quite low for large LLMs.

Deepseek R1 (full, Q1) is 14t/s on an M2 Ultra, so this should be around 20t/s

Re: Apple M3 Ultra

#557

Post by rbanffy »

agloe_dreams wrote: post #206
lauritz wrote: post #46

They update the Studio to M3 Ultra now, so M4 Ultra can presumably go directly into the Mac Pro at WWDC? Interesting timing. Maybe they'll change the form factor of the Mac Pro, too? Additionally, I would assume this is a very low-volume product, so it being on N3B isn't a dealbreaker. At the same time, these chips must be very expensive to make, so tying them with luxury-priced RAM makes some kind of sense.

My understanding was that Apple wanted to figure out how to build systems with multi-SOCs to replace the Ultra chips. The way it is currently done means that the Max chips need to be designed around the interconnect. Theoretically speaking, a multi-SOC setup could also scale beyond two chips and into a wider set of products.

Ultra is already two big M3 chips coupled through an interposer. Apple is curiously not going the way of chiplets like the big CPU crowd is.

Re: Apple M3 Ultra

#558

Post by thesuitonym »

hnaccount_rng wrote: post #383
naikrovek wrote: post #231

Earlier quoted context omitted.

> But Apple is also not interested in that market, so this will probably never happen. they're just a tiny company with shareholders who are really tired of never earning back their investments. give 'em a break. I mean they're still so small that they must protect themselves by requiring that macs be used for publishing iPhone and iPad applications.

Not to get in the way of good snark or anything. But.. Apple isn't _requiring_ that everyone uses MacOS on their systems. But you have to bring your own engineering effort to actually make another OS run. And so far Asahi is the only effort that I'm aware of (there were alternatives in the very beginning, but they didn't even get to M2 right?)

> But you have to bring your own engineering effort to actually make another OS run.

I mean, that's usually how it works though. When IBM launched the PS/2, they didn't support anything other than PC-DOS and OS/2, Microsoft had to make MS-DOS work for it (I mean... they did get support from IBM, but not really), the 386BSD and Linux communities brought the engineering effort without IBM's involvement.

When Apple was making Motorola Macs, they may have given Be a little help, but didn't support any other OSes that appeared. Same with PowerPC.

All of the support for alternative OSes has always come from the community, whether that's volunteers or a commercial interest with cash to burn. Why should that change for Apple silicon?

Re: Apple M3 Ultra

#559

Post by bangaladore »

zitterbewegung wrote: post #545
PeterStuer wrote: post #518

Earlier quoted context omitted.

Is this on chip memory? From the 800GB/s I would guess more likely a 512bit bus (8 channel) to DDR5 modules. Doing it on a quad channel would just about be possible, but really be pushing the envelope. Still a nice thing. As for practicality, which mainstream applications would benefit from this much memory paired with a nice but relative mid compute? At this price-point (14K for a full specced system), would you pre…

NVIDIA project DIGITS has 128 GB LPDDR5x coherent unified system memory at a 273 Gb/s memory bus speed.

It would be 273 GB/s (gigabytes, not gigabits). But in reality we don't know the bandwidth. Some ex employee said 500 GB/s.

You're source is a reddit post in which they try to match the size to existing chips, without realizing that its very likely that NVIDIA is using custom memory here produced by Micron. Like Apple uses custom memory chips.

Re: Apple M3 Ultra

#560

Post by lhl »

FloatArtifact wrote: post #301
cxie wrote: post #222

512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…

They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigabytes of uRAM for 10 grand. The the question is if a llm will run with usable performance at that scale? The point is there's diminishing returns despite having enough uRAM with the same amount of memory bandwidth even with increased processing speed of the new…

Since no one specifically answered your question yet, yes, you should be able to get usable performance. A Q4_K_M GGUF of DeepSeek-R1 is 404GB. This is a 671B MoE that "only" has 37B activations per pass. You'd probably expect in the ballpark of 20-30 tok/s (depends on how much actually MBW can be utilized) for text generation.

From my napkin math, the M3 Ultra TFLOPs is still relatively low (around 43 FP16 TFLOPs?), but it should be more than enough to handle bs=1 token generation (should be way <10 FLOPs/byte for inference). Now as far is its prefill/prompt processing speed... well, that's another matter.

Post reply on HN