Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apple M3 Ultra

apple.com

Post reply on HN
Print view View original

641–650 of 1001 posts

Re: Apple M3 Ultra

#641

Post by bick_nyers »

justincormack wrote: post #626
dragonwriter wrote: post #541

Earlier quoted context omitted.

128GB for 3K; per the announcement their ConnectX networking allows two Project Digits devices to be plugged into eachother and work together as one device giving you 256GB for $6k, and, AFAIK, existing frameworks can split models across devices, as well, hence, presumably, the upthread suggestion that Project Digits would provide 512GB for $12k, though arguably the last step is cheating.

the reason Nvidia only talk about two machines over the network is I think they only have one network port, so you need to add costs for a switch.

If you want to split tensorwise yes. Layerwise splits could go over Ethernet.

I would be interested to see how feasible hybrid approaches would be, e.g. connect each pair up directly via ConnectX and then connect the sets together via Ethernet.

Re: Apple M3 Ultra

#642

Post by kridsdale1 »

rjeli wrote: post #312

Wow, incredible. I told myself I’d stop waffling and just buy the next 800gb/s mini or studio to come out, so I guess I’m getting this. Not sure how much storage to get. I was floating the idea of getting less storage, and hooking it up to a TB5 NAS array of 2.5” SSDs, 10-20tb for models + datasets + my media library would be nice. Any recommendations for the best enclosure for that?

It depends on your bandwidth needs.

I also want to build the thing you want. There are no multi SSD M2 TB5 bays. I made one that holds 4 drives (16TB) at TB3 and even there the underlying drives are far faster than the cable.

My stuff is in OWC Express 4M2.

Re: Apple M3 Ultra

#643

Post by klausa »

sroussey wrote: post #612
klausa wrote: post #297

Earlier quoted context omitted.

Baseline M4 doesn't have Thunderbolt 5 either; only the Pro/Max variants do. The press-release even calls TB5 out: >Each Thunderbolt 5 port is supported by its own custom-designed controller directly on the chip. Given that they're doing the same on A-series chips (A18 Pro with 10Gbps USB-C; A18 with USB 2.0); I imagine it's just relatively simple to swap the I/O blocks around and they're doing this for cost and/or p…

Which means this is a whole new chip. It may be M3 based, but with added interposer support and new thunderbolt stuff. Which, at this point, why not just use M4 as a base?

>Which, at this point, why not just use M4 as a base?

I imagine that making those chips is quite a bit more involved than just taking the files for M3 Max, and copy-pasting them twice into a new project.

I imagine it just takes more time to design/verify/produce them; especially given they're not selling very many of them, so they're probably not super-high-priority projects.

Re: Apple M3 Ultra

#644

Post by adrian_b »

aurareturn wrote: post #546
adrian_b wrote: post #516

Earlier quoted context omitted.

In a dual-socket EPYC system, the memory bandwidth is higher than in this Apple system by 40% (i.e. 1152 GB/s), and the memory capacity can be many times higher. Like another poster said, 768 GB of ECC RDIMM DDR5-6000 costs around $5000. Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple s…

Any program whose performance is limited by memory bandwidth, as it can be frequently the case for inference, will run significantly faster in such an EPYC server than in the Apple system, even when running on the CPU. Source on this? CPUs would be very compute constrained.

According to Apple, the GPU of M3 Ultra has 80 graphics cores, which should mean 10240 FP32 execution units, the same like an NVIDIA RTX 4080 Super.

However Apple does not say anything about the GPU clock frequency, which I assume that it is significantly less than that of NVIDIA.

In comparison, a dual-socket AMD Turin can have up to 12288 FP32 execution units, i.e. 20% more than an Apple GPU.

Moreover, the clock frequency of the AMD CPU must be much higher than that of the Apple GPU, so it is likely that the AMD system may be at least twice faster for computing some graphic application than the Apple M3 Ultra GPU.

I do not know what facilities exist in the Apple GPU for accelerating the computations with low-precision data types, like the tensor cores of NVIDIA GPUs.

While for graphic applications big server CPUs are actually less compute constrained than almost all consumer GPUs (except RTX 4090/5090), the GPUs can be faster for ML/AI applications that use low-precision data types, but this is not at all certain for the Apple GPU.

Even if the Apple GPU happens to be faster for some low-precision data type, the difference cannot be great.

However a server that would beat the Apple M3 Ultra GPU computationally would cost much more than $10k, because it would need CPUs with many cores.

If the goal is only to have a system with 50% more memory and 40% more memory bandwidth than the Apple system, that can be done at a $10k price.

While such a system would become compute constrained more often than an Apple GPU, it would still beat it every time when the memory would be the bottleneck.

Re: Apple M3 Ultra

#645

Post by woadwarrior01 »

titzer wrote: post #466
hangonhn wrote: post #457

Earlier quoted context omitted.

Do we know if is it slower because of hardware is not as well suited for the task or is it mostly a software issue -- the code hasn't been optimized to run on Apple Silicon?

AFAICT the neural engine has accelerators for CNNs and integer math, but not the exact tensor operations in popular LLM transformer architectures that are well-supported in GPUs.

The neural engine is perfectly capable of accelerating matmults. It's just that autoregressive decoding in single batch LLM inference is memory bandwidth constrained, so there are no performance benefits to using the ANE for LLM inference (although, there's a huge power efficiency benefit). And the only way to use the neural engine is via CoreML. Using the GPU with MLX or MPS is often easier.

Re: Apple M3 Ultra

#646

Post by joshhart »

This is pretty exciting. Now an organization could produce an open weights mixture of experts model that has 8-15b active parameters but could still be 500b+ parameters and it could be run locally with INT4 quantization with very fast performance. DeepSeek R1 is a similar model but over 30b active parameters which makes it a little slow.

I do not have a good sense of how well quality scales with narrow MoEs but even if we get something like Llama 3.3 70b in quality at only 8b active parameters people could do a ton locally.

Re: Apple M3 Ultra

#649

Post by sgt »

aurareturn wrote: post #562
sgt wrote: post #543

Earlier quoted context omitted.

What kind of Nvidia-based rig would one need to achieve 40 tokens/sec on Deepseek 671b? And how much would it cost?

Around 5x Nvidia A100 80GB can fit 671b Q4. $50k just for the GPUs and likely much more when including cooling, power, motherboard, CPU, system RAM, etc.

So the M3 Ultra is amazing value then. And from what I could tell, an equivalent AMD Epyc would still be so constrained that we're talking 4-5 tokens/s. Is this a fair assumption?

Re: Apple M3 Ultra

#650

Post by jorams »

hnaccount_rng wrote: post #383
naikrovek wrote: post #231

Earlier quoted context omitted.

> But Apple is also not interested in that market, so this will probably never happen. they're just a tiny company with shareholders who are really tired of never earning back their investments. give 'em a break. I mean they're still so small that they must protect themselves by requiring that macs be used for publishing iPhone and iPad applications.

Not to get in the way of good snark or anything. But.. Apple isn't _requiring_ that everyone uses MacOS on their systems. But you have to bring your own engineering effort to actually make another OS run. And so far Asahi is the only effort that I'm aware of (there were alternatives in the very beginning, but they didn't even get to M2 right?)

Note that they said (emphasis mine):

> they're still so small that they must protect themselves by requiring that macs be used for publishing iPhone and iPad applications.

They're not talking about Apple's silicon as a target, but as a development platform.

Post reply on HN