I wouldn't buy any Mac products for LLM usage. It's cheaper to just buy a subscription. You enjoy Mac for your personal/professional use. I have been running LLM on Mac, honestly for such an expensive machine I wouldn't want to use it for long runs just for a quick 30 minute usage. It heats up so much you can make an omelette on it. I would NOT want my dedicated MacBook run hot for long these things last forever. Fairly confident if I run LLM's all day long their life will be short lived. In my family my kids are still using their laptops which are papas handovers. I have an iMac from 2014 that my kids still use.
Ask HN: MacBook vs. Dedicated GPU for LLM
71–76 of 76 posts
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#72MacBooks with their unified memory behave like a slow GPU with enormous amount of video RAM. So you can run large smart models slowly. Dedicated GPUs have less video RAM so can run smaller less smart models quickly.
Unified memory is also a kicker for crashing a Mac.
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#73This multi-model unified gateway addresses a huge pain point of juggling separate billing dashboards and rate limits across dozens of LLMs. The complete LICENSE and contribution docs make extension very approachable for external developers
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#74I think nothing beats graphics cards—you can always add one or upgrade to a newer model.
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#75Earlier quoted context omitted.
Latest MBP goes up to 128GB of memory.
The M5 Max's GPU is slower than a single RTX 3090, though. Both machines will be stuck in the 30-50GB model range, but the 3090 would have faster token prefill and faster decode speeds (614GB/s on M5 Max vs 936GB/s on 1x 3090).
> Both machines will be stuck in the 30-50GB model range,
That is not true.
3090 is only 24GB.
It will absolutely outperform the M5 on models that will fit in that space, but M5 will outperform with larger models as the whole 128GB is GPU addressable.
Re: Ask HN: MacBook vs. Dedicated GPU for LLM
#76How bad is the token-per-second drop-off on the Mac once you scale up the context window? Does it hit a wall at 32K or 64k tokens?