From personal experience - it works, but you won't get a comfortable time to first token (latency is high). The reason is that prefill on Macs is bad. You need to have a lot more cores to do it quick. It's close to instant for small models on NVidia GPU's but on Macs it takes a few seconds to get the answer for a simple prompt. And the time grows proportionally with your context size.
Ask HN: MacBook vs. Dedicated GPU for LLM
61–70 of 76 posts
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#62MacBooks with their unified memory behave like a slow GPU with enormous amount of video RAM. So you can run large smart models slowly. Dedicated GPUs have less video RAM so can run smaller less smart models quickly.
Do Mac Pros provide more headroom? noob here, noob questions
Not that much - a bit better but still negligible. Prefill is highly parallelizable and benefits from many cores. NVidia GPU's simply have several times more cores than even Pros.
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#63Earlier quoted context omitted.
So the MacBook Neo costs the same as a decent bike? What makes you think the Ultra will be in that range as well?
I wouldn’t race on my commuter bike or work professionally on a MacBook Neo. I guess it could start around 3000€
I think you're still making my point that comparing to "a decent bike" means nothing. It depends on what the bike will be used for, and what "decent" might mean to the person interpreting it.
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#64Earlier quoted context omitted.
I agree. To run an acceptable model (e.g. Qwen/Qwen3.6-27B or google/gemma-4-31B) with a good quantization (minimum Q5) with a good context size (min 64k) you could buy 2 or even 3 GTX 5060 16GiB VRAM for ~550$ each. Fyi the much faster MoE models were useless for my usecases - e.g not able to correctly identify me/I/you, endless thinking loops, etc. I'm currently running those models using an RTX 5070 12GiB + RTX 50…
how many tokens per second do you get?
Not the OP, but their setup must be faster than my 4060 16GB + 3060 12GB setup. Here are my numbers (typical values, N=1):
Model pp (t/s) tg (t/s)
Qwen 3.6 27B 900 29
Qwen 3.6 35B-A3B 2100 85
Gemma 4 31B 750 28
Gemma 4 26B-A4B 2500 90
- Layer splitting used. Tensor splitting is ~1.2x faster in TG, but power spikes from 150W to 380W.
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#65It depends. I have a M5 128 so i can play around with large models and even keep several of them loaded at once and use something like llama swap to access them all via bifrost or litellm. You won't do this without some serious local GPU's with big memory. The downside is the speed, it's not fast, but fast enough to tinker and develop with worrying about ongoing cloud cost. When done developing and you need to really scale up this is when you can swap to cloud computing and get the job done faster. My $5k macbook can do more than a $50k nvidia/intel/amd setup, just not as fast.
So you need to decide whats important to you if you want to work locally, large/many models or speed. It's the pick 2 problem speed, size, cost, pick 2 or go with cloud and accept your development time is also spent $$ on each iteration.
I was hoping for the M5 ultra by now, but looks like that's not coming until much later this year for a much higher price now.
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#66I asked a few of my friends that are ML engineers this question and all of them said to run the LLMs in the cloud with their infrastructure because it was going to be way faster. If you just want to tinker around I would look at @JSR_FDD's comment.
My dad avoided buying a home PC for a long time, because he felt the systems he used at work in the 80s were so much more powerful than anything for the home at the time, that he didn’t see a point.
I can’t help but draw parallels here.
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#67I asked a few of my friends that are ML engineers this question and all of them said to run the LLMs in the cloud with their infrastructure because it was going to be way faster. If you just want to tinker around I would look at @JSR_FDD's comment.
My dad avoided buying a home PC for a long time, because he felt the systems he used at work in the 80s were so much more powerful than anything for the home at the time, that he didn’t see a point. I can’t help but draw parallels here.
And he wouldn't be wrong for that era. Prices also dropped pretty quickly while specs would almost double within a span of months. Whatever he could've bought in Jan would be easily outclassed by anything 12 months later.
We don't have that anymore. Specs have more or less stabilised and what you're buying now could easily last you years.
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#68[dead]
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#69Earlier quoted context omitted.
I agree. To run an acceptable model (e.g. Qwen/Qwen3.6-27B or google/gemma-4-31B) with a good quantization (minimum Q5) with a good context size (min 64k) you could buy 2 or even 3 GTX 5060 16GiB VRAM for ~550$ each. Fyi the much faster MoE models were useless for my usecases - e.g not able to correctly identify me/I/you, endless thinking loops, etc. I'm currently running those models using an RTX 5070 12GiB + RTX 50…
How are you running these together, splitting the model somehow or did you mean different models on any one card at a time?
I always run just one model at once, I switch between them depending on what I do (e.g. Qwen3.6 27B Q5_K_M by Bartowski when programming with "OpenHands" and when doing OCR text transcription and transformation, Gemma4 31B-it Q5_K_M by Bartowski when chatting in Open WebUI doing general tasks).
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#70Earlier quoted context omitted.
how many tokens per second do you get?
Not the OP, but their setup must be faster than my 4060 16GB + 3060 12GB setup. Here are my numbers (typical values, N=1): Model pp (t/s) tg (t/s) Qwen 3.6 27B 900 29 Qwen 3.6 35B-A3B 2100 85 Gemma 4 31B 750 28 Gemma 4 26B-A4B 2500 90 - All models: UD-Q4 w/ MTP. Context size: ~100k (MoE) / ~70k (Dense). - Layer splitting used. Tensor splitting is ~1.2x faster in TG, but power spikes from 150W to 380W.
In my case it somehow depends a lot on the task being performed... .
E.g. when doing text transcription/OCR from images (Qwen 3.6 27B Q4_K_M by Bartowski) with a context size of ~50k I get a pp of ~460 tokens per second and a generation ranging from 35 to 45 tokens per second (using "--spec-type draft-mtp --spec-draft-n-max 2" currently with llama.cpp b6548).
On the other hand when handling code (Qwen 3.6 27B Q5_K_M by Bartowski) with a context size of 128k I get a pp ranging between 500 to 1500 tokens per second and a generation between 25 and 40 tokens per second (using in this case as well "--spec-type draft-mtp --spec-draft-n-max 2" currently with llama.cpp b6548).
Anyway in theory with "--split-mode layer" I think that it's anyway the slowest card that drives the overall performance (I do see in "nvtop" that usually the 5070 is ~25% active, the 5060 ~50% and the 3060 ~75%).