Ask HN: Anyone Using a Mac Studio for Local AI/LLM?

I have a maxed out M3 Ultra. It runs quantized large open Chinese models pretty well. It's slow-ish, but since I don't use them very frequently, most of the time is waiting to the model to load from disk to RAM.

There are benchmarks on token generation speed out there for some of the large models. You can probably guess the speed for models you're interested in by comparing the sizes (mostly look at the active params).

Currently the main issue for M1-M4 is the prompt "preprocessing" speed. In practical terms, if you have a very long prompt, it's going to take a much long time to process it. IIRC it's due to lack of efficient matrix multiplication operations in the hardware, which I hear is rectified in the M5 architecture. So if you need to process long prompts, don't count on the Mac Studio, at least not with large models.

So in short, if your prompts are relatively short (eg. a couple thousand tokens at most), you need/want a large model, you don't need too much scale/speed, and you need to run inference locally, then Macs are a reasonable option.

For me personally, I got my M3 Ultra somewhat due to geopolitical issues. I'm barred from accessing some of the SOTA models from the US due to where I live, and sometimes the Chinese models are not conveniently accessible either. With the hardware, they can pry DeepSeek R1, Kimi-K2, etc. from my cold dead hands lol.

Below are my test results after running local LLMs on two machines.

I'm using LM Studio now for ease of use and simple logging/viewing of previous conversations. Later I'm gonna use my own custom local LLM system on the Mac Studio, probably orchestrated by LangChain and running models with llama.cpp.

My goal has all the time been to use them in ensembles in order to reduce model biases. The same principle has just now been introduced as a feature called "model council" in Perplexity Max: https://www.perplexity.ai/hub/blog/introducing-model-council

Chats will be stored in and recalled from a PostgreSQL database with extensions for vectors (pgvector) and graph (Apache AGE).

For both sets of tests below, MLX was used when available, but ultimately ran at almost the same speed as GGUF.

I hope this information helps someone!

/////////

Mac Studio M3 Ultra (default w/96 GB RAM, 1 TB SSD, 28C CPU, 60C GPU):

• Gemma 3 27B (Q4_K_M): ~30 tok/s, TTFT ~0.52 s

• GPT-OSS 20B: ~150 tok/s

• GPT-OSS 120B: ~23 tok/s, TTFT ~2.3 s

• Qwen3 14B (Q6_K): ~47 tok/s, TTFT ~0.35 s

(GPT-OSS quants and 20B TTFT info not available anymore)

//////////

MacBook Pro M1 Max 16.2" (64 GB RAM, 2 TB SSD, 10C CPU, 32C GPU):

• Gemma 3 1B (Q4_K): ~85.7 tok/s, TTFT ~0.39 s

• Gemma 3 27B (Q8_0): ~7.5 tok/s, TTFT ~3.11 s

• GPT-OSS 20B (8bit): ~38.4 tok/s, TTFT ~21.15 s

• LFM2 1.2B: ~119.9 tok/s, TTFT ~0.57 s

• LFM2 2.6B (Q6_K): ~69.3 tok/s, TTFT ~0.14 s

• Olmo 3 32B Think: ~11.0 tok/s, TTFT ~22.12 s

can only speak for myself here, but the prompt processing speeds on Apple Sillicon is too slow, especially for any meaningful usage

My experience with Mac Studio is that memory bandwidth matters more than raw cores for reasonable LLM throughput locally; curious what others find for models >13B parameters?

The M5 ultra series is supposed to have some big gains around prompt processing - something like 3-4x from what I've read. I'm tempted to swap out my m4 mini that I'm using for this kind of stuff right now!