Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My local model setup on an M4 Pro Mac Mini

lws.io

Post reply on HN
Print view View original

141–150 of 207 posts

Re: My local model setup on an M4 Pro Mac Mini

#141

Post by icedchai »

taylorhou wrote: post #77
amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests. GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent…

This may work for your use case, but sounds abysmally slow for any complex coding task.

Re: My local model setup on an M4 Pro Mac Mini

#142

Post by vintagedave »

taylorhou wrote: post #77
amanzi wrote: post #2

No mention of the performance of the models? I'm able to load a bunch of different models on my little mini-PC with 16GB RAM, but the performance is terrible. I always wonder what performance people are getting with local models that they find is acceptable?

i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests. GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent…

So this is something like a $10,000 machine before RAM prices rose?

I see Apple is currently selling a 256GB M5 for about $10K, so buying October's 512GB one could be, what, $13-14K?

A $0 per-token bill is great but this is clearly not something for normal people, just some businesses.

Re: My local model setup on an M4 Pro Mac Mini

#143

Post by helsinkiandrew »

BoredomIsFun wrote: post #111
visarga wrote: post #93

Earlier quoted context omitted.

Local is never cheaper than cloud because they can do batch inference, and that means you load model weights once to produce 128 tokens on 128 sessions in parallel not 1 token on 1 session like local models. Local models rarely get to high utilization factor, they spend most of their time waiting. If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models.

> not 1 token on 1 session like local models. Local models can absolutely run in batch, what are even talking about? > If you had only batch inference and enough of it to fill the compute to 80% then you get cheaper local models. Even if you ran sequentally, single session, a _finetuned_ tiny (8B) local model on narrow tasks would abolutely mog SOTAs, any of it - Fable, Opus, Sol you name it.

> Local models can absolutely run in batch, what are even talking about?

I think the point was that if you aren't running your local machine at 100% for 24 hours a day then a cloud - with multiple clients - that is, will be more efficient.

Re: My local model setup on an M4 Pro Mac Mini

#144

Post by bel8 »

LeBit wrote: post #113
NamlchakKhandro wrote: post #105

Earlier quoted context omitted.

48gb of vram. a machine like this is about a years rent for most people. a small car for most others.

I think he’s talking about the Mac Mini unified memory. 48G RAM is pretty useful if you want to run k8s locally for tests / exploration

true but if you're actually running k8s and similar workloads, chances are it might eat memory that LLM requires.

you'll also notice these articles rarely specify their context window in tokens, because it is small, usually 30k to 70k tokens and it gets slower as it fills up.

Re: My local model setup on an M4 Pro Mac Mini

#145

Post by wila »

pornel wrote: post #140
wila wrote: post #127

Is there anything one can reasonably run on a mac mini M2 with just 24GB RAM or should I not even try?

Not enough for coding. 48GB is minimum for a non-lobotomized coding model like qwen, and you'd likely want 64GB to have long context and not kernel panic when Chrome opens. You could run one of the smaller Gemma models to have a chatty Wikipedia.

Figured that would be the case. Thanks.

Re: My local model setup on an M4 Pro Mac Mini

#147

Post by wila »

ch_sm wrote: post #129
wila wrote: post #127

Is there anything one can reasonably run on a mac mini M2 with just 24GB RAM or should I not even try?

It depends on your use case, but the smaller Gemma 4 models or qwen3.6:9b would probably run OK on that. I recommend trying it, even just for fun. It‘s easy with omlx.

Ok, will see if I can find some time for play. Thanks.

Re: My local model setup on an M4 Pro Mac Mini

#148

Post by bel8 »

vintagedave wrote: post #142
taylorhou wrote: post #77

Earlier quoted context omitted.

i have a 512gb ram m3 ultra mac studio setup with a gas city that runs one of my companies. today was the first time ever that a local model (GLM5.3 8-bit) was able to match fable5 in our tests. GLM-5.3-Flash at true 8-bit: 341 GB on disk, 328 GB resident, 288 experts across 46 layers, loads in 65 seconds. • 18.7 tokens/s generation, 35 tokens/s prompt, on a desk, on a $0 per-token bill. • Runs beside our whole agent…

So this is something like a $10,000 machine before RAM prices rose? I see Apple is currently selling a 256GB M5 for about $10K, so buying October's 512GB one could be, what, $13-14K? A $0 per-token bill is great but this is clearly not something for normal people, just some businesses.

Props to your parent commenter for including context size. Because 131k context window is prohibitively small for my coding workloads so I know a 512GB Mac won't cut it.

Re: My local model setup on an M4 Pro Mac Mini

#149

Post by bel8 »

ericd wrote: post #21
gruez wrote: post #18

Earlier quoted context omitted.

>I honestly wouldn’t bother with local models right now unless I either had a 5090 and was happy with running Qwen 3.8 27B How's the actual performance of Qwen 3.8 27B? On deepswe it supposedly performs slightly worse than gpt 5.6 luna high[1], but I can't help but think they've been benchmaxxed. [1] https://deepswe.datacurve.ai/ , https://unsloth.ai/docs/models/qwen3.8#benchmarks

Not sure, I haven't run it, I've just been running DS V4 Flash non-stop since it came out, and that's replaced a lot of my Claude Code usage. People seem very impressed, though, it seems like it trades vram/world knowledge for extra thinking time, which I think is a good trade for local. tbf, I've heard luna's not great at coding. Fast and good for things like classifiers, summarization, though. A friend and I were a…

> We were speculating that maybe it's because a lot of benches are best-of-n, and should probably be worst-of-n.

Thank you. I just changed my opinion on this thanks to you. I agree now, since we tend to execute LLM tasks once instead of N times anyway.

I suspect models like Fable executes the same task N times in parallel and picks best answer or merges them to for a better answer.

Re: My local model setup on an M4 Pro Mac Mini

#150

Post by bel8 »

Aurornis wrote: post #74
ux266478 wrote: post #54

Earlier quoted context omitted.

> Pretty expensive is an understatement. [...] If you could it would be multiple hundreds of thousands of dollars. Obviously, I quantified both the operating expense and the capital expense in my post. What I find curious is that you're quoting me talking about the operating expenditure, and changing the topic to be about the buy-in like these are interchangeable things. You don't think that this is a crucial and imp…

> You could have spent all of 5 seconds of searching rather than just assuming[1]. I guarantee this will not ship to you any time soon. The current lead time on these GPUs in measured in years. If you didn't place an order for this a long time ago, it's not coming this year. Being able to add it to an online configurator does not mean anything right now. > 12 months of Claude burning $70k a month is $840k Your math i…

It usually boils down to people trying to convince themselves that keeping their macs hot and with very little ram to spare only to get sub 50 tokens per second on a subpar lobotomized (quantized) model is worth it.

And I'm not even considering their time spent fiddling, fine tuning configs to adjust for ram, updating/benchmarking models, etc. Which is probably more expensive than the mac so the math is even more wrong.

Post reply on HN