Curious to know your experience running local LLM's with a well spec'ed out M3 Ultra or M4 Pro Mac Studio. I don't see a lot of discussion on the Mac Studio for Local LLMs but it seems like you could put big models in memory with the shared VRAM. I assume that the token generation would be slow, but you might get higher quality results because you can put larger models in memory.
Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
1–10 of 39 posts
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#2There are some people on r/LocalLlama using it [0]. Seems like the consensus is while it does have more unified RAM for running models, up to half a terabyte, the token generation speed can be fairly slow such that it might just be better to get an Nvidia or AMD machine.
[0] https://old.reddit.com/r/LocalLLaMA/search?q=mac+studio&rest...
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#3I do! I have an M3 Ultra with 512GB. A couple of opencode sessions running work well. Currently running GML 4.7 but was on Kimi K2.5. Both great. Excited for more efficiencies to make their way to LLMs in general.
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#4Mine is a M1 ultra with 128gb of ram. It's fast enough for me.
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#5Not a Mac Studio but I use a basic Macbook Pro laptop with 24 GB of RAM (16 usable as VRAM) and I can run a number of models on it at decent speed, my main bottleneck is context window size, but if I am asking single purpose questions I am fine.
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#6I do! I have an M3 Ultra with 512GB. A couple of opencode sessions running work well. Currently running GML 4.7 but was on Kimi K2.5. Both great. Excited for more efficiencies to make their way to LLMs in general.
Excellent. Thanks for the info!
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#7There are some people on r/LocalLlama using it [0]. Seems like the consensus is while it does have more unified RAM for running models, up to half a terabyte, the token generation speed can be fairly slow such that it might just be better to get an Nvidia or AMD machine. [0] https://old.reddit.com/r/LocalLLaMA/search?q=mac+studio&rest...
Thanks for the link. I'll take a look.
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#8I do! I have an M3 Ultra with 512GB. A couple of opencode sessions running work well. Currently running GML 4.7 but was on Kimi K2.5. Both great. Excited for more efficiencies to make their way to LLMs in general.
How's the inference speed? What was the price? I'm guessing you can fit the entire model without quantization?
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#9Not a Mac Studio but I use a basic Macbook Pro laptop with 24 GB of RAM (16 usable as VRAM) and I can run a number of models on it at decent speed, my main bottleneck is context window size, but if I am asking single purpose questions I am fine.
Yeah. I'm currently on an Mac Mini m2 Pro with 32GB or ram, and I was so curious how much more I could get out of the Apple ecosystem. Thanks for your perspective.
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#10Mine is a M1 ultra with 128gb of ram. It's fast enough for me.
Thanks for the perspective!