Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My local model setup on an M4 Pro Mac Mini

lws.io

Post reply on HN
Print view View original

41–50 of 208 posts

Re: My local model setup on an M4 Pro Mac Mini

#41

Post by srcreigh »

Gigachad wrote: post #16
pcarolan wrote: post #4

Earlier quoted context omitted.

It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.

It does make me wonder how the hosted stuff is so cheap. For pretty much everything else, hosted/rented is more expensive but offers better convenience and flexibility. But for AI, even if you consider the total lifetime cost and are utilizing it heavily. You never break even by buying.

They're not cheap at all. I did one xhigh Qwen 3.8 27B agentic coding task last week via OpenRouter and it cost me like $10.

99% of the cost was in input tokens, I only used like 100k ish output tokens. It was a one shot task asking the agent to implement proxy injection to Guice. It did a pretty amazing job.

If you were to use hosted LLMs for a lot of agentic coding, a maxed out M5 Ultra Mac Studio would pay for itself in under a year.

Re: My local model setup on an M4 Pro Mac Mini

#42

Post by brainless »

I experiment a lot with local LLMs, particularly small ones like Qwen3.5 4B and 9B. I have build multiple experiments to make harnesses that use these models for code generation, planning, local search, etc.

These are really good models but the harness has to be built around them. I have a ton of generated system prompts for specific purposes. Even parts of a SolidJS stack, for example Route management, has its own prompt. These are experiments but the results are real. If we build harnesses around small models, we can build a locally running WYSIWYG editor which works on plain text prompts.

The performance, in simple tokens/second, is not the most important factor. For many private data points, like emails, I would rather have a local graph based search and LLM on top where the harness is specific to problems like calendar, contacts, finance, etc.

I run all experiments on an 16GB M4 Mac Mini but coding agents building the harness are a mix of Codex, Claude Code and opencode.

Re: My local model setup on an M4 Pro Mac Mini

#43

Post by hkchad »

brettdav wrote: post #39
hkchad wrote: post #5

Earlier quoted context omitted.

I run a similar setup to the one he described on similar hardware. I run bifrost and llama swap though (tailscale rocks). My local model usage is for some out of band batch processing one of my personal apps uses. Basically a personalized recommender for media, it curates stuff for me based on a database i've compiled over years, so non-interactive. For that use case, I don't really care that it might take a few minu…

Can you share a bit more about your bifrost and llama swap setup? I’m facing memory constraints and am looking for a managed model solution that will help with hot swapping loaded models and stay-warm concurrency. Ideally with prioritization.

What do you want to know? Just start llama-swap with the models i have downloaded, add llama-swap as a provider in bifrost, expose the models you want and they become available in one single endpoint you can use in anything like opencode, openwebui or anything that speaks openai.

Re: My local model setup on an M4 Pro Mac Mini

#44

Post by ericd »

Aurornis wrote: post #33
ux266478 wrote: post #24

Earlier quoted context omitted.

That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week. With an 8x MI…

> With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars. > It does mean 8 multi-trillion parameter models unquantized runnin…

>You couldn’t buy one of these if you wanted to right now.

You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 . You can get thousands of tps of GLM 5.3 output out of this thing, which grades around Opus 4.8. Payoff is around 1 year vs. spot prices on these GPUs, including power.

Re: My local model setup on an M4 Pro Mac Mini

#45

Post by CamperBob2 »

ericd wrote: post #44
Aurornis wrote: post #33

Earlier quoted context omitted.

> With an 8x MI355x cluster at full tilt and including cooling, your power draw runs ~17kW. That's what it looks like when it's running full tilt. To be fair, hey that's pretty expensive. Pretty expensive is an understatement. You couldn’t buy one of these if you wanted to right now. If you could it would be multiple hundreds of thousands of dollars. > It does mean 8 multi-trillion parameter models unquantized runnin…

>You couldn’t buy one of these if you wanted to right now. You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 . You can get thousands of tps of GLM 5.3 output out of this thing, which grades around Opus 4.8. Payoff is around 1 year vs. spot prices on these GPUs, including power.

I can't tell from the ad -- it says "supports" 8x MI350X GPUs, but does that mean "includes" 8x MI350X GPUs? For $300K I'd certainly hope so, but I'm assuming not.

A system with 4x RTX 6000s costs about $60K these days, and can (as you note) trade blows with Opus 4.8 if not Fable. In fact, it'll give you a better pelican than Fable 5.1, and in less time.

Re: My local model setup on an M4 Pro Mac Mini

#46

Post by jumploops »

c0rruptbytes wrote: post #14
jumploops wrote: post #9

My biggest problem with running local LLMs on my M4 Max/128GB RAM is the prefill latency. I've since acquired two DGX Sparks, and it feels so much snappier.

m5 max really fixed pp with the better matmul support, im sure the m5 ultra will be even crazier the sparks have much slower memory bandwidth is the trade off

I believe the dgx spark is still twice as fast at prefill as the m5 max, but the ultra should get closer to parity.

Another benefit of the 2x spark setup is that you can parallelize to ~6 streams pretty efficiently.

All depends on the workflows you’re using it for.

I’m quite excited for the M7 class machines.

Re: My local model setup on an M4 Pro Mac Mini

#47

Post by pcarolan »

ux266478 wrote: post #24
pcarolan wrote: post #4

Earlier quoted context omitted.

It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.

That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week. With an 8x MI…

Here’s an experiment: purchase an anthropic pro max subscription for $200/m. Now go buy the hardware to run DeepSeek’s equivalent. In a year, who spent more?

Re: My local model setup on an M4 Pro Mac Mini

#48

Post by ericd »

CamperBob2 wrote: post #45
ericd wrote: post #44

Earlier quoted context omitted.

>You couldn’t buy one of these if you wanted to right now. You can: https://www.exxactcorp.com/Exxact-TS4-149591758-E149591758 . You can get thousands of tps of GLM 5.3 output out of this thing, which grades around Opus 4.8. Payoff is around 1 year vs. spot prices on these GPUs, including power.

I can't tell from the ad -- it says "supports" 8x MI350X GPUs, but does that mean "includes" 8x MI350X GPUs? For $300K I'd certainly hope so, but I'm assuming not. A system with 4x RTX 6000s costs about $60K these days, and can (as you note) trade blows with Opus 4.8 if not Fable. In fact, it'll give you a better pelican than Fable 5.1, and in less time.

Ha fair, I'd definitely confirm with a salesperson before wiring them $300k. But most of the signs on the configurator seem to point to it including the GPUs? Not going to make 30k BTUs/hr of heat without the 8kw of GPUs.

Re: My local model setup on an M4 Pro Mac Mini

#49

Post by ericd »

ux266478 wrote: post #24
pcarolan wrote: post #4

Earlier quoted context omitted.

It’s not. Do it as a hobby or for privacy but for performance just use a frontier model api. You’re paying less than cost for something that would take tens of thousands to set up locally.

That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week. With an 8x MI…

What're you using that monster for?

Re: My local model setup on an M4 Pro Mac Mini

#50

Post by ericd »

pcarolan wrote: post #47
ux266478 wrote: post #24

Earlier quoted context omitted.

That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week. With an 8x MI…

Here’s an experiment: purchase an anthropic pro max subscription for $200/m. Now go buy the hardware to run DeepSeek’s equivalent. In a year, who spent more?

That's not apples to apples on almost any dimension.
Post reply on HN