My local model setup on an M4 Pro Mac Mini

Post by chlorion » Wed, Sep 02, 2026, 10:03 AM UTC

SXX wrote: post #124

srcreigh wrote: post #61

Earlier quoted context omitted.

It’s not so clear after 5 years that you’ll come out ahead. You’ll have spent $20k. The apple computer owner will probably be running local models that are better than today’s frontier on the same hardware. Idk where you live, but where I am running the M5 Ultra Mac Studio at max rated power 24/7 for a month costs C$42. The considerations against Apple hardware are 1) hardware advancements 2) early access to the best…

> The apple computer owner will probably be running local models that are better than today’s frontier on the same hardware. Hardware is not magically getting more memory or bandwidth. Believing there will be some magical optimizations to compensate for it is just dellusion.

Post by phazonoverload » Wed, Sep 02, 2026, 10:05 AM UTC

amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

Post by phazonoverload » Wed, Sep 02, 2026, 10:07 AM UTC

c16 wrote: post #94

> Running a large model locally comes down to one thing: how much RAM it actually needs in memory. Not completely true. It's memory AND memory bandwidth. You can have 1tb of memory but if you have awful memory-bandwidth you'll also have slow tok/s. A3B helps with this, but so does MTP. From my experience, you'd be better off running the dense 27b-mlx with MTP than the 3.6 version with A3B. You say your model is ~20GB…

Post by anotherCodder » Wed, Sep 02, 2026, 10:20 AM UTC

srcreigh wrote: post #41

Gigachad wrote: post #16

Earlier quoted context omitted.

It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.

They're not cheap at all. I did one xhigh Qwen 3.8 27B agentic coding task last week via OpenRouter and it cost me like $10. 99% of the cost was in input tokens, I only used like 100k ish output tokens. It was a one shot task asking the agent to implement proxy injection to Guice. It did a pretty amazing job. If you were to use hosted LLMs for a lot of agentic coding, a maxed out M5 Ultra Mac Studio would pay for its…

Post by icedchai » Wed, Sep 02, 2026, 10:25 AM UTC

Normal_gaussian wrote: post #126

taylorhou wrote: post #77

Earlier quoted context omitted.

i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests. GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent…

$0 per-token bill You still have electricity and capital investment. Envelope math suggests cheap electricity is costing you something like $0.50/mtok and the opportunity cost on the capital tied up and lost in the unit purchase and resale is going to cost you something like $2/mtok at 100% utilization (so, frontier model prices or higher at real utilization), and you don't benefit from any elasticity. Hosted GLM 5.3…