I have a m3 ultra 256gb and it’s okay. Some coworkers find it too slow, but they also use cerebras…
Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
31–39 of 39 posts
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#32Would matmul acceleration in m5 do much to improve token Generation? I’m planning to upgrade when the new studio updates to m5 but if it’s not much of a shift I’ll probably look at a second hand 256+ ram m3 studio.
Edit: of course the software would need to leverage the neural accelerators which is another variable if the software supports it in the first place
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#33I'm using an M3 Ultra w/ 512GB of RAM, using LMStudio and mostly mlx models. It runs massive models with reasonable tokens per second, though prompt processing can be slow. It handles long conversations fine so long as the KV cache hits. It's usable with opencode and crush, though my main motivation for getting it was specifically to be able to process personal data (e.g. emails) privately, and to experiment freely w…
>trying to figure out ... fast external storage Acasis makes 40gbps external nVME cases. Mine feels quick (for non-LLM tasks). I also use 10gbps Terramaster 4-bay RAIDs (how I finally retired my Pro5,1). >energy usage This thing uses an order of magnitude -less- energy than the computer it replaced, and is faster in almost every aspect.
10gbps is slow enough to be annoying when you're loading a 200GB model, unfortunately.
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#34Earlier quoted context omitted.
>trying to figure out ... fast external storage Acasis makes 40gbps external nVME cases. Mine feels quick (for non-LLM tasks). I also use 10gbps Terramaster 4-bay RAIDs (how I finally retired my Pro5,1). >energy usage This thing uses an order of magnitude -less- energy than the computer it replaced, and is faster in almost every aspect.
10gbps is slow enough to be annoying when you're loading a 200GB model, unfortunately.
You might consider then getting four 40gbps nVME enclosures, and then RAIDing multiple together (e.g. in a big stripe, you could get 160gbps throughput, only limited by # physical interfaces). Each slice could be +TBs.
Obviously increases your failure rate, but if you're constantly updating the same models (and not creating your own) you don't really need redundancy.
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#35I'm using an M3 Ultra w/ 512GB of RAM, using LMStudio and mostly mlx models. It runs massive models with reasonable tokens per second, though prompt processing can be slow. It handles long conversations fine so long as the KV cache hits. It's usable with opencode and crush, though my main motivation for getting it was specifically to be able to process personal data (e.g. emails) privately, and to experiment freely w…
[dead]
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#36I went looking for the latest line of apple computers after reading this thread and I noticed they force you into the higher CPU's in order to get the higher amounts of unified memory.
So not only are they content charging +$400 or +$600 for RAM which in itself ludicrously overpriced, they force you to upgrade +$1000-2000 on the top CPU's.
Its impossible to spec a macbook pro or a mac mini with a base CPU and a decent amount of RAM. Total scam since they know people want the RAM to use with local LLMs.
This was not always the case - When I specced out my macbook pro M1 16gb it was entirely possible to get 32 and 64gb without any tie-in to CPU upgrades.
I was ready to drop a few grand on a new macbook pro M5 or M4 pro with a decent amount of RAM but it's currently set up to be an insane price gouge.
To get 32GB of RAM it's an M5 chip price $1999.
To get 64GB of RAM you are forced to to grab the M4 max CPU, and it's $3,899 on apple right now. What a scam.
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#37I went looking for the latest line of apple computers after reading this thread and I noticed they force you into the higher CPU's in order to get the higher amounts of unified memory. So not only are they content charging +$400 or +$600 for RAM which in itself ludicrously overpriced, they force you to upgrade +$1000-2000 on the top CPU's. Its impossible to spec a macbook pro or a mac mini with a base CPU and a decen…
They're just limiting the range of SKUs they have to manufacture. For all we know, the base M-series die might not even support that larger amount of in-package memory to begin with.
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#38[dead]
Re: Ask HN: Anyone Using a Mac Studio for Local AI/LLM?
#39[dead]