Apple M3 Ultra • Hacker News Board

Re: Apple M3 Ultra

#301

Post by FloatArtifact » Wed, Mar 05, 2025, 3:51 PM UTC

cxie wrote: post #222

512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…

They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigabytes of uRAM for 10 grand.

The the question is if a llm will run with usable performance at that scale? The point is there's diminishing returns despite having enough uRAM with the same amount of memory bandwidth even with increased processing speed of the new chip for AI.

So there must be a min-max performance ratio between memory bandwidth and the size of the memory pool in relation to the processing power.

Re: Apple M3 Ultra

#302

Post by fintechie » Wed, Mar 05, 2025, 3:51 PM UTC

IMO this is a bigger blow to the AI big boys than Deepseek's release. This is massive for local inference. Exciting times ahead for open source AI.

Re: Apple M3 Ultra

#303

Post by jread » Wed, Mar 05, 2025, 3:53 PM UTC

AlchemistCamp wrote: post #197

InTheArena wrote: post #29

Whoa. M3 instead of M4. I wonder if this was basically binning, but I thought that I had read somewhere that the interposer that enabled this for the M1 chips where not available. That Said, 512GB of unified ram with access to the NPU is absolutely a game changer. My guess is that Apple developed this chip for their internal AI efforts, and are now at the point where they are releasing it publicly for others to use.…

Keep in mind the minimum configuration that has 512GB of unified RAM is $9,499.

$8549 with 1TB storage

Re: Apple M3 Ultra

#304

Post by whimsicalism » Wed, Mar 05, 2025, 3:53 PM UTC

seabrookmx wrote: post #271

criddell wrote: post #226

Earlier quoted context omitted.

The Asahi maintainer resigned recently. What that means for the future only time will tell. I probably wouldn't want to make a big investment in it right now.

Your wording makes it sound like it was a one-man show. Asahi has a really strong contributor base, new leadership[1], and the backing of Fedora via the Asahi Fedora Remix. While Hector resigning is a loss, I don't think it's a death knell for the project. [1]: https://asahilinux.org/2025/02/passing-the-torch/

it was pretty close to a one man show

Re: Apple M3 Ultra

#305

Post by aurareturn » Wed, Mar 05, 2025, 3:53 PM UTC

whimsicalism wrote: post #260

spiderfarmer wrote: post #255

Earlier quoted context omitted.

You can use Thunderbolt 5 interconnect (80Gbps) to run LLMs distributed across 4 or 5 Mac Studios.

why would you ever want to do that remains an open question

Probably some kind of local LLM server. 1TB of 1.6 TB/s memory if you link 2 together. $20k total. Half the price of a single Blackwell chip.

Re: Apple M3 Ultra

#306

Post by hedora » Wed, Mar 05, 2025, 3:53 PM UTC

ryao wrote: post #240

stego-tech wrote: post #228

Earlier quoted context omitted.

> This hardware is really being held back by the operating system at this point. It really is. Even if they themselves won't bring back their old XServe OS variant, I'd really appreciate it if they at least partnered with a Linux or BSD (good callout, ryao) dev to bring a server OS to the hardware stack. The consumer OS, while still better (to my subjective tastes) than Windows, is increasingly hampered by bloat and…

Given shared heritage, I would expect to see Apple work with FreeBSD before I would expect Apple to work with Linux.

I heard OpenBSD has been working for a while.

I’m continually surprised Apple doesn’t just donate something like 0.1% of their software development budget to proton and the asahi projects. It’d give them a big chunk of the gaming and server markets pretty much overnight.

I guess they’re too busy adding dark patterns that re-enable siri and apple intelligence instead.

Re: Apple M3 Ultra

#307

Post by dagmx » Wed, Mar 05, 2025, 3:53 PM UTC

NorwegianDude wrote: post #282

dagmx wrote: post #163

Earlier quoted context omitted.

You're comparing two different things. The compute level you’re talking about on the M3 Ultra is the neural engine. Not including the GPU. I expect the GPU here will be behind a 5090 for compute but not by the unrelated numbers you’re quoting. After all, the 5090 alone is multiple times the wattage of this SoC.

No, I'm not. I'm comparing the TOPS of the M3 Ultra and the tensor cores of the RTX 5090. If not, what is the TOPS of the GPU, and why isn't apple talking about it if there is more performance hidden somewhere? Apple states 18 TOPS for the M3 Max. And why do you think Apple added the neural engine, if not to accelerate compute? The power draw is quite a bit higher, but it's still much more efficient as the performanc…

The ANE and tensor cores are not comparable though. One is literally meant for low cost inference while the others are meant for acceleration of training.

If you squint, yeah they look the same, but so does the microcontroller on the GPU and a full blown CPU. They’re fundamentally different purposes, architectures and scale of use.

The ANE can’t even really be used directly. Apple heavily restricts the use via CoreML APIs for inference. It’s only usable for smaller, lightweight models.

If you’re comparing to the tensor cores, you really need to compare against the GPU which is what gets used by apples ml frameworks such as MLX for training etc.

It will still be behind the NVIDIA GpU, but not by anywhere near the same numbers.

Re: Apple M3 Ultra

#308

Post by Gracana » Wed, Mar 05, 2025, 3:54 PM UTC

whimsicalism wrote: post #278

gatienboquet wrote: post #78

Earlier quoted context omitted.

LLMs are primarily "memory-bound" rather than "compute-bound" during normal use. The model weights (billions of parameters) must be loaded into memory before you can use them. Think of it like this: Even with a very fast chef (powerful CPU/GPU), if your kitchen counter (VRAM) is too small to lay out all the ingredients, cooking becomes inefficient or impossible. Processing power still matters for speed once everythin…

Transformers are typically memory- bandwidth bound during decoding. This chip is going to have a much worse memory b/w than the nvidia chips. My guess is that these chips could be compute-bound though given how little compute capacity they have.

It's pretty close. A 3090 or 4090 has about 1TB/s of memory bandwidth, while the top Apple chips have a bit over 800GB/s. Where you'll see a big difference is in prompt processing. Without the compute power of a pile of GPUs, chewing through long prompts, code, documents etc is going to be slower.

Re: Apple M3 Ultra

#309

Post by whimsicalism » Wed, Mar 05, 2025, 3:54 PM UTC

pmarreck wrote: post #200

InTheArena wrote: post #92

Earlier quoted context omitted.

No native docker support, no headless management options (enterprise strength), Limited QoS management, lack of robust python support (out of the box), interactive user focused security model.

> lack of robust python support There is no such thing. Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers would you use? And how would you prevent "project collision" (where one Python project bumps into another one and one just stops working)? Example: SSL library differences across projects is a notorious culprit. Python is garbage and I don't understand w…

these are solved problems now, check back in. uv is now the standard

Re: Apple M3 Ultra

#310

Post by aurareturn » Wed, Mar 05, 2025, 3:54 PM UTC

hedora wrote: post #287

exabrial wrote: post #203

Earlier quoted context omitted.

If Apple supported Linux (headless) natively, and we could rack m4 pros, I absolutely would use them in our Colo. The CPUs have zero competition in terms of speed, memory bandwidth. Still blown away no other company has been able to produce Arm server chips that can compete.

The last I checked, AMD was outperforming Apple perf/dollar on the high end, though they were close on perf/watt for the TDPs where their parts overlapped. I’d be curious to know if this changes that. It’d take a lot more than doubling cores to take out the very high power AMD parts, but this might squeeze them a bit. Interestingly, AMD has also been investing heavily in unified RAM. I wonder if they have / plan an S…

The M4 Pro is 56% faster in ST performance against AMD’s new Strix Halo while being 3.6x more efficient.

Source: https://www.notebookcheck.net/AMD-Ryzen-AI-Max-395-Analysis-...

Cinebench 2024 results.