Apple M3 Ultra • Hacker News Board

Re: Apple M3 Ultra

#451

Post by vlovich123 » Wed, Mar 05, 2025, 5:18 PM UTC

Espressosaurus wrote: post #413

vlovich123 wrote: post #91

Earlier quoted context omitted.

No, with limited VRAM you could offload the model partially or split across CPU and GPU. And since CPU has swap, you could run the absolute largest model. It’s just really really slow.

The difference between Deepseek-r1:70b (edit: actually 32b) running on an M4 Pro (48 GB unified RAM, 14 CPU cores, 20 GPU cores) and on an AMD box (64 GB DDR4, 16 core 5950X, RTX 3080 with 10 GB of RAM) is more than a factor of 2. The M4 pro was able to answer the test prompt twice--once on battery and once on mains power--before the AMD box was able to finish processing. The M4's prompt parsing took significantly lo…

You're adding detail that's not relevant to anything I said. I was saying this statement:

> VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis.

Is false. Regardless of how much VRAM you have, if the criteria is "can run even if slowly", all machines can run all models because you have swap. It's unusably slow but that's not what OP was claiming the difference is.

Re: Apple M3 Ultra

#452

Post by sunshowers » Wed, Mar 05, 2025, 5:18 PM UTC

dghlsakjg wrote: post #161

dwedge wrote: post #146

Earlier quoted context omitted.

I've been buying and using MBP for 6 or 7 years now, and just assumed I could run Linux on one if I wanted to. I just spent a couple of days trying to get a 2018 MBP working with Linux and found out [edit to clarify] that my other ARM MBP basically won't work. I just want a break from MacOS, I'll be buying a Thinkpad and will probably never come back. This isn't my moaning, I understand it's their market, but if thei…

A 2018 MacBook would be an intel x86 chip. It’s incredibly easy to get Linux running on that machine.

No, there's a bunch of MBP generations in the middle that just never got any Linux attention.

Re: Apple M3 Ultra

#453

Post by matt-p » Wed, Mar 05, 2025, 5:18 PM UTC

dheera wrote: post #375

cxie wrote: post #222

512GB of unified memory is truly breaking new ground. I was wondering when Apple would overcome memory constraints, and now we're seeing a half-terabyte level of unified memory. This is incredibly practical for running large AI models locally ("600 billion parameters"), and Apple's approach of integrating this much efficient memory on a single chip is fascinating compared to NVIDIA's solutions. I'm curious about how…

It will cost 4X what it costs to get 512GB on an x86 server motherboard.

Not really like for like.

The pricing isn't as insane as you'd think, 96 to 256GB is 1500 which isn't 'cheap' but, it could be worse.

All in 5,500 gets you a ultra with 256GB memory, 28 cores, 60 GPU cores, 10Gb network - I think you'd be hard pushed to build a server for less.

Re: Apple M3 Ultra

#454

Post by tomn » Wed, Mar 05, 2025, 5:21 PM UTC

pmarreck wrote: post #200

InTheArena wrote: post #92

Earlier quoted context omitted.

No native docker support, no headless management options (enterprise strength), Limited QoS management, lack of robust python support (out of the box), interactive user focused security model.

> lack of robust python support There is no such thing. Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers would you use? And how would you prevent "project collision" (where one Python project bumps into another one and one just stops working)? Example: SSL library differences across projects is a notorious culprit. Python is garbage and I don't understand w…

This is incoherent to me. Your complaints are about packaging, but the elixir wrapper doesn't deal with that in any way -- it just wraps UV, which you could use without elixir.

What am I missing?

Also, typically when people say things like

> Tell me, which combination of the 15+ virtual environments, dependency management and Python version managers

It means they have been trapped in a cycle of thinking "just one more tool will surely solve my problem", instead of realising that the tools _are_ the problem, and if you just use the official methods (virtualenv and pip from a stock python install), things mostly just work.

Re: Apple M3 Ultra

#455

Post by _zoltan_ » Wed, Mar 05, 2025, 5:23 PM UTC

qwertox wrote: post #52

iambateman wrote: post #14

People who know more than me: they’re talking a lot about RAM and not much about GPU. Do you expect this will be able to handle AI workloads well? All I’ve heard for the past two years is how important a beefy GPU is. Curious if that holds true here too.

A beefy GPU which can't hold models in VRAM is of very limited use. You'll see 16 GB of VRAM on gamer Nvidia cards, the RTX 5090 being an exception with 32 GB VRAM. The professional cards have around 96 GB of VRAM. The thing with these Apple chips is that they have unified memory, where CPU and GPU use the same memory chips, which means that you can load huge models into RAM (no longer VRAM, because that doesn't exis…

which professional card has 96GB of VRAM?

Re: Apple M3 Ultra

#456

Post by _zoltan_ » Wed, Mar 05, 2025, 5:25 PM UTC

whimsicalism wrote: post #321

Gracana wrote: post #308

Earlier quoted context omitted.

It's pretty close. A 3090 or 4090 has about 1TB/s of memory bandwidth, while the top Apple chips have a bit over 800GB/s. Where you'll see a big difference is in prompt processing. Without the compute power of a pile of GPUs, chewing through long prompts, code, documents etc is going to be slower.

nobody in industry is using a 4090, they are using H100s which have 3TB/s. Apple also doesn’t have any equivalent to nvlink. I agree that compute is likely to become the bottleneck for these new Apple chips, given they only have like ~0.1% the number of flops

4.8TB/s on H200, 8TB/s on B200, pretty insane.

Re: Apple M3 Ultra

#457

Post by hangonhn » Wed, Mar 05, 2025, 5:30 PM UTC

diggan wrote: post #393

FloatArtifact wrote: post #301

Earlier quoted context omitted.

They didn't increase the memory bandwidth. You can get the same memory bandwidth, which is available on the M2 Studio. Yes, yes, of course you can get 512 gigabytes of uRAM for 10 grand. The the question is if a llm will run with usable performance at that scale? The point is there's diminishing returns despite having enough uRAM with the same amount of memory bandwidth even with increased processing speed of the new…

> The the question is if a llm will run with usable performance at that scale? This is the big question to have answered. Many people claim Apple can now reliably be used as a ML workstation, but from the numbers I've seen from benchmarks, the models may fit in memory, but the performance for tok/sec is so slow to not feel worth it, compared to running it on NVIDIA hardware. Although it be expensive as hell to get 51…

Do we know if is it slower because of hardware is not as well suited for the task or is it mostly a software issue -- the code hasn't been optimized to run on Apple Silicon?

Re: Apple M3 Ultra

#458

Post by bick_nyers » Wed, Mar 05, 2025, 5:30 PM UTC

valine wrote: post #408

FloatArtifact wrote: post #392

Earlier quoted context omitted.

> Probably helps that models like deepseek are mixture of expert. Having all weights in VRAM means you don’t have to unlod/reload. Memory bandwidth usage should be limited to the 37B active parameters. "Memory bandwidth usage should be limited to the 37B active parameters." Can someone do a deep dive above quote. I understand having the entire model loaded into RAM helps with response times. However, I don't quite un…

With a mixture of experts model you only need to read a subset of the weights from memory to compute the output of each layer. The hidden dimensions are usually smaller as well so that reduces the size of the tensors you write to memory.

Just to add onto this point, you expect different experts to be activated for every token, so not having all of the weights in fast memory can still be quite slow as you need to load/unload memory every token.

Re: Apple M3 Ultra

#459

Post by raydev » Wed, Mar 05, 2025, 5:31 PM UTC

I know it's basically nitpicking competing luxury sports cars at this point, but I am very bothered that existing benchmarks for the M3 show single core perf that is approximately 70% of M4 single core perf.

I feel like I should be able to spend all my money to both get the fastest single core performance AND all the cores and available memory, but Apple has decided that we need to downgrade to "go wide". Annoying.

Re: Apple M3 Ultra

#460

Post by Espressosaurus » Wed, Mar 05, 2025, 5:31 PM UTC

vlovich123 wrote: post #451

Espressosaurus wrote: post #413

Earlier quoted context omitted.

The difference between Deepseek-r1:70b (edit: actually 32b) running on an M4 Pro (48 GB unified RAM, 14 CPU cores, 20 GPU cores) and on an AMD box (64 GB DDR4, 16 core 5950X, RTX 3080 with 10 GB of RAM) is more than a factor of 2. The M4 pro was able to answer the test prompt twice--once on battery and once on mains power--before the AMD box was able to finish processing. The M4's prompt parsing took significantly lo…

You're adding detail that's not relevant to anything I said. I was saying this statement: > VRAM is what takes a model from "can not run at all" to "can run" (even if slowly), hence the emphasis. Is false. Regardless of how much VRAM you have, if the criteria is "can run even if slowly", all machines can run all models because you have swap. It's unusably slow but that's not what OP was claiming the difference is.

The criteria for purchase for anybody trying to use it is "run slowly but acceptably" vs. "run so slow as to be unusable".

My memory is wrong, it was the 32b. I'm running the 70b against a similar prompt and the 5950X is probably going to take over an hour for what the M4 managed in about 7 minutes.

edit: an hour later and the 5950 isn't even done thinking yet. Token generation is generously around 1 token/s.

edit edit: final statistics. M4 Pro managing 4 tokens/s prompt eval, 4.8 tokens/s token generation. 5950X managing 150 tokens/s prompt eval, and 1 token/s generation.

Perceptually I can live with the M4's performance. It's a set prompt, do something else, come back sort of thing. The 5950/RTX3080's is too slow to be even remotely usable with the 70b parameter model.