Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My local model setup on an M4 Pro Mac Mini

lws.io

Post reply on HN
Print view View original

21–30 of 207 posts

Re: My local model setup on an M4 Pro Mac Mini

#21

Post by ericd »

gruez wrote: post #18
ericd wrote: post #15

Earlier quoted context omitted.

I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code,…

>I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed. [1] https://deepswe.datacurve.ai/ , https://unsloth.ai/docs/models/qwen3.8#benchmarks

Not sure, I haven't run it, I've just been running DS V4 Flash non-stop since it came out, and that's replaced a lot of my Claude Code usage. People seem very impressed, though, it seems like it trades vram/world knowledge for extra thinking time, which I think is a good trade for local. tbf, I've heard luna's not great at coding. Fast and good for things like classifiers, summarization, though.

A friend and I were actually discussing today how benches show Luna Max at about par on coding with Sol Medium, but how it's nowhere near in reality. We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n, because variance in performance is killer with large coding projects. Consistency is what lets you actually build on this stuff.

Re: My local model setup on an M4 Pro Mac Mini

#22

Post by whatsThisBtn4 »

miles_io wrote: post #12

M4 Pro has been a solid performer for iterating on smaller local models. Much more convenient than spinning up cloud instances for dev.

If you just want chat.

Agents require at least DeepSeek pro and even that is the minimum.

You might be able to get a good model to write instructions and run it in smaller models.

Otherwise, cool your AI got the current weather.

Re: My local model setup on an M4 Pro Mac Mini

#24

Post by ux266478 »

pcarolan wrote: post #4
amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.

That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week.

With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. It does mean 8 multi-trillion parameter models unquantized running 24/7 without pause. And you get the full month like that, your monthly token limit is the time in a month. That cluster, the electrical upgrade, the cooling setup, and the electricity to run it all costs less in 2 months than your maximum affordance from Anthropic does in the same time period. Two billing cycles, and realistically it's more like two weeks. In 4 quarters you've wasted over a million. Like, what are we talking about here?

Now if you aren't using AI all that much, which is perfectly valid, and especially if you aren't using it at its absolute maximum, the story changes. Because even though at that point you're not paying nearly as much in electricity to run the cluster anymore, you still have the $300k+ capex to get the setup in the first place. But if we're not redlining it non-stop, then we're not really talking about performance anymore, are we? If your org never comes close to hitting token limits, it's probably because AI is rather marginal for you. Which again, is perfectly valid. I don't even use AI professionally.

Fact of the matter is, if your corp can justify the capex for a cluster and makes heavy use of AI, you are literally burning money by not having one in your building. The numbers are painfully obvious. Even deepseek isn't as cheap. This is before we get into things like LoRAs, custom inference pipelines, etc. which you know are kind of important if you actually care about model performance.

Re: My local model setup on an M4 Pro Mac Mini

#25

Post by ramgine »

whatsThisBtn4 wrote: post #19
amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

I can't imagine using CPU... Oh I did twice. If you are work from home and do dishes between prompts you can get a gpt3-like result. I found it useful when I was... Well I didn't find it useful. But an Nvidia 3060 let me ask unethical questions pretty fast.

With which model. I have a 3060 with a bunch of system ram

Re: My local model setup on an M4 Pro Mac Mini

#26

Post by villish »

gruez wrote: post #18
ericd wrote: post #15

Earlier quoted context omitted.

I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code,…

>I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed. [1] https://deepswe.datacurve.ai/ , https://unsloth.ai/docs/models/qwen3.8#benchmarks

Keep in mind these downloadable models use 3-10x the amount of tokens as well. You really can’t beat a couple $20 subscriptions.

https://quesma.com/benchmarks/babaisbench/

Re: My local model setup on an M4 Pro Mac Mini

#27

Post by cdnsteve »

amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

These are estimates: https://tokenstead.ai/find/results?hardware_id=23&use_case=c...

Re: My local model setup on an M4 Pro Mac Mini

#29

Post by shell0x »

jumploops wrote: post #9

My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency. I've since acquired two DGX Sparks, and it feels so much snappier.

Would you mind sharing your local Mac setup and which models you currently use and whether it’s GGUF or MLX? I’ve the hardware same specs.

Re: My local model setup on an M4 Pro Mac Mini

#30

Post by ericd »

villish wrote: post #26
gruez wrote: post #18

Earlier quoted context omitted.

>I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed. [1] https://deepswe.datacurve.ai/ , https://unsloth.ai/docs/models/qwen3.8#benchmarks

Keep in mind these downloadable models use 3-10x the amount of tokens as well. You really can’t beat a couple $20 subscriptions. https://quesma.com/benchmarks/babaisbench/

Looks like GLM 5.2 is coming in at Great showing from Sol, though.

But also, it's Baba Is You :-D

Post reply on HN