Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My local model setup on an M4 Pro Mac Mini

lws.io

Post reply on HN
Print view View original

11–20 of 207 posts

Re: My local model setup on an M4 Pro Mac Mini

#11

Post by arcanemachiner »

mintflow wrote: post #10

Have a macmini m4 32G, not the pro version, previously everytime I tried local LLM is a bit disappointing, and I finally decide to not waste time and perhaps in the future invest a better hardware to server more modern and dense model I am curious is what is the 80% request served by this setup, I was using it for OpenClaw which run serveral cron jobs that discover stuffs over the wide internet, check my support syst…

You have tried Qwen 3.8 27B before coming to this conclusion, I hope? It's an incremental improvement over 3.6, but I mostly want to make sure you didn't just try running some old junker before coming to this conclusion.

Re: My local model setup on an M4 Pro Mac Mini

#14

Post by c0rruptbytes »

jumploops wrote: post #9

My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency. I've since acquired two DGX Sparks, and it feels so much snappier.

m5 max really fixed pp with the better matmul support, im sure the m5 ultra will be even crazier

the sparks have much slower memory bandwidth is the trade off

Re: My local model setup on an M4 Pro Mac Mini

#15

Post by ericd »

amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code, which somewhat makes up for it being dumber.

That said, it is really cool to be able to run an LLM on eg a Mac laptop. Just not a better experience on almost any metric for interactive use than eg Claude Code, beside privacy and guardrails.

Re: My local model setup on an M4 Pro Mac Mini

#16

Post by Gigachad »

pcarolan wrote: post #4
amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.

It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.

Re: My local model setup on an M4 Pro Mac Mini

#17

Post by api »

Gigachad wrote: post #16
pcarolan wrote: post #4

Earlier quoted context omitted.

It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.

It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.

There are economies of scale but there’s also a data center bubble (probably) so there might be some selling dollars for fifty cents going on.

Re: My local model setup on an M4 Pro Mac Mini

#18

Post by gruez »

ericd wrote: post #15
amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B, or a pair of DGX Sparks running DSv4 flash, or better, 2x6000 RTX Blackwells. Those are the kinds of rigs that the local model enthusiasts are running. With the GPU setups, you’re looking at generally >100tps generation in single stream, and >10k tps of prefill, so it’s snappier than Claude code,…

>I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B

How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed.

[1] https://deepswe.datacurve.ai/, https://unsloth.ai/docs/models/qwen3.8#benchmarks

Re: My local model setup on an M4 Pro Mac Mini

#19

Post by whatsThisBtn4 »

amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

I can't imagine using CPU... Oh I did twice.

If you are work from home and do dishes between prompts you can get a gpt3-like result.

I found it useful when I was... Well I didn't find it useful. But an Nvidia 3060 let me ask unethical questions pretty fast.

Re: My local model setup on an M4 Pro Mac Mini

#20

Post by whatsThisBtn4 »

pcarolan wrote: post #4
amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.

I watched someone at a fortune 20 company get embarrassed for buying a Mac to run a 70B model in 2025.

He was a lead engineer, so after he announced it wasn't going to work, everyone pretended it never happened. But we all knew.

Post reply on HN