Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My local model setup on an M4 Pro Mac Mini

lws.io

Post reply on HN
Print view View original

61–70 of 208 posts

Re: My local model setup on an M4 Pro Mac Mini

#61

Post by srcreigh »

pcarolan wrote: post #47
ux266478 wrote: post #24

Earlier quoted context omitted.

That's not even remotely close to being true, even once you account for capex. You have to look at the actual usage, look at the token limits. Even if you're paying Anthropic $200k/month for scale-tier, you're going to blow through your token limits trying to run max output 24/7. Three users running Opus 4.8 at max non-stop will probably clean your monthly allowance from daddy Dario in less than a week. With an 8x MI…

Here’s an experiment: purchase an anthropic pro max subscription for $200/m. Now go buy the hardware to run DeepSeek’s equivalent. In a year, who spent more?

It’s not so clear after 5 years that you’ll come out ahead. You’ll have spent $20k. The apple computer owner will probably be running local models that are better than today’s frontier on the same hardware.

Idk where you live, but where I am running the M5 Ultra Mac Studio at max rated power 24/7 for a month costs C$42.

The considerations against Apple hardware are 1) hardware advancements 2) early access to the best models. But it’s really not that clear.

(The other guy who thought hosted models on openrouter are cheap has spent $100k in 5 years.)

Re: My local model setup on an M4 Pro Mac Mini

#64

Post by ttul »

Most people running local models would probably love to run larger models if only they had access to big enough hardware. I'm curious: to those of you running models locally, if there was a way to inference the model of your choice at a reasonable cost by effectively time-sharing a B300 rack through some privacy-protecting intermediary, would you consider that?

If there was a "Mullvad of GPU clouds", would that solve the privacy concerns?

Re: My local model setup on an M4 Pro Mac Mini

#65

Post by ux266478 »

Aurornis wrote: post #52
CamperBob2 wrote: post #45

Earlier quoted context omitted.

I can't tell from the ad -- it says "supports" 8x MI350X GPUs, but does that mean "includes" 8x MI350X GPUs? For $300K I'd certainly hope so, but I'm assuming not. A system with 4x RTX 6000s costs about $60K these days, and can (as you note) trade blows with Opus 4.8 if not Fable. In fact, it'll give you a better pelican than Fable 5.1, and in less time.

> trade blows with Opus 4.8 if not Fable. Okay I love the open models, but the hype is getting ridiculous. The models you can run on 4 X RTX6000 are not Fable level.

Baseline yeah. But part of the reason you run open models is how much nicer fine tuning them is. Granted, you probably don't want to try and make LoRAs on a 4x RTX6000 setup, but you could if you really wanted to and there are other ways to modify models. And yes, if you're good at it, you can turn a piddly mid-range model that's only good at benchmarks into a heavyweight clanker (for a specific domain).

Re: My local model setup on an M4 Pro Mac Mini

#66

Post by argee »

dolebirchwood wrote: post #53
argee wrote: post #3

Earlier quoted context omitted.

I have an M4 pro (48 GB ram) and I run Gemma 4 26b a4b at 52 tok/s and Qwen 3.5b a3b at 72 tok/s. Both 4bit quantized. These are enough for my needs and the performance is more than good enough. I'm not running the MLX version of the Gemma model, if I did the inference speed would likely be a bit better. I wouldn't use them for coding features though.

> enough for my needs Which are...?

Some examples (keep in mind this is all indefinitely free for me, no burning quota away):

1. Getting information (such as information about hardware unfamiliar to me) when not connected to the internet, which happens occasionally in my case.

2. Continuing to learn Rust by way of toy examples, puzzles, and comparing aspects of various solutions, for example from LeetCode.

3. Reformatting data, for example from a PDF to a markdown table, or converting receipt images to text.

4. Simple translation/explanation (e.g. I'm teaching my wife one of the languages I speak but sometimes may not know/have the words to explain the full nuance of a translated word).

5. Summarization. One of the webnovels I'm reading has some very boring parts I don't want to slog through, in those cases I simply make the LLM summarize that part and move on.

Etc., you get the idea. It's not unusable for coding, but it would make many mistakes when making a whole feature and the context lengths are limited to around 30k-40k tokens by my RAM. I could give it access to the web but I simply use an online model when I need that sort of thing, again partly due to the context limit.

Edit: The MLX version of Gemma 4 26b a4b does about 62 tok/s.

Re: My local model setup on an M4 Pro Mac Mini

#67

Post by arjie »

ericd wrote: post #60
arjie wrote: post #57

Earlier quoted context omitted.

I have quoted large nodes from this supplier and have lots of^W^W GPUs from them for personal use. Current lead time is more than 30 months. They're a good provider but you have to be a big shot buying NVL72s before you're getting anything within your payback period.

Ah thanks for the solid info, too bad. I'd seen them come up as a pretty good price for 6000 RTX's in the past, which seem generally pretty available, good source for those?

Yeah, they're good source. But the price for those GPUs is 5 figs even with the nvidia startup program nowadays. Also, I went back and looked. Most of my GPUs are actually from Central Computers who were great, but Exxact is real too. So "lots of" was inaccurate.

Also, the lead time I quoted was for individual 8x nodes.

Re: My local model setup on an M4 Pro Mac Mini

#68

Post by ericd »

arjie wrote: post #67
ericd wrote: post #60

Earlier quoted context omitted.

Ah thanks for the solid info, too bad. I'd seen them come up as a pretty good price for 6000 RTX's in the past, which seem generally pretty available, good source for those?

Yeah, they're good source. But the price for those GPUs is 5 figs even with the nvidia startup program nowadays. Also, I went back and looked. Most of my GPUs are actually from Central Computers who were great, but Exxact is real too. So "lots of" was inaccurate. Also, the lead time I quoted was for individual 8x nodes.

Ah yeah, one of mine is from Central. And yeah, crazy how much they've gone up. But I can see why, they scream.

Re: My local model setup on an M4 Pro Mac Mini

#69

Post by gigatexal »

I really like these show and tell style posts. I’m always curious how people have their setups and what tools they use. Also the blog has a nice theme and is easy to read.

I wanna get a desktop Mac for local ai so that I don’t turn my laptop into a delta 15k rpm fan when I run things.

I guess I’ll get in line for one hah.

Re: My local model setup on an M4 Pro Mac Mini

#70

Post by srcreigh »

Gigachad wrote: post #58
srcreigh wrote: post #41

Earlier quoted context omitted.

They're not cheap at all. I did one xhigh Qwen 3.8 27B agentic coding task last week via OpenRouter and it cost me like $10. 99% of the cost was in input tokens, I only used like 100k ish output tokens. It was a one shot task asking the agent to implement proxy injection to Guice. It did a pretty amazing job. If you were to use hosted LLMs for a lot of agentic coding, a maxed out M5 Ultra Mac Studio would pay for its…

Qwen is weirdly expensive. Deepseek v4 flash is dirt cheap. You'd need at least 128gb of ram to run this model and in my experience, a days work with it costs around 80 cents.

So I ran the math, assuming the agent takes 75 turns per 200k context, with deepseek v4 flash it costs around $2.57 to reach 1M context in 375 turns. Cached input costs scale quadratically with # of agent turns.

Considering that I hit the 1M compaction multiple times per day with codex, it would definitely cost at least $5-8/day to use deepseek how I normally use codex.

Post reply on HN