Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models on an M4 with 24GB memory

jola.dev

Post reply on HN
Print view View original

51–60 of 189 posts

Re: Running local models on an M4 with 24GB memory

#51

Post by brcmthrowaway »

guessmyname wrote: post #26
nu11ptr wrote: post #17

Still trying to understand if a Macbook Pro M5 Max with 128GB is likely going to be able to run coding models well enough that I can cancel my Codex, or even go down to the $20/month plan.

A 128GiB MacBook Pro in Canada is what, north of CAD $11k after tax? That’s around USD $7k. At $20/month for a cloud AI subscription, you’re looking at almost 30 years of service for the same money. How long do people realistically expect a laptop to stay competitive with SOTA local models? Especially in a space where model sizes, context windows, and inference requirements keep moving every year. And even if the har…

This is one of the best takedowns of local models I've ever seen.

I just hate paying money for cloud subscriptions, and work has given me a decent laptop

Re: Running local models on an M4 with 24GB memory

#53

Post by knollimar »

guessmyname wrote: post #26
nu11ptr wrote: post #17

Still trying to understand if a Macbook Pro M5 Max with 128GB is likely going to be able to run coding models well enough that I can cancel my Codex, or even go down to the $20/month plan.

A 128GiB MacBook Pro in Canada is what, north of CAD $11k after tax? That’s around USD $7k. At $20/month for a cloud AI subscription, you’re looking at almost 30 years of service for the same money. How long do people realistically expect a laptop to stay competitive with SOTA local models? Especially in a space where model sizes, context windows, and inference requirements keep moving every year. And even if the har…

You have to use the item a lot, to the point where you'd be exceeding subs a lot

Re: Running local models on an M4 with 24GB memory

#54

Post by quacker »

layoric wrote: post #50
quacker wrote: post #37

I could have used this article before I spent the weekend arriving to the same conclusion! Same laptop, and my contrived test was having it fix 50 or so lint errors in a small vibe-coded C++ repo. I wanted it to be able to handle a bunch of small tasks without getting stuck too often. GPT OSS 20B was usable but slow, and actually frequently made mistakes like adding or duplicating statements unnecessarily, listing th…

Honestly surprised to hear that GPT OSS 20B runs slow on mac hardware. It's absolutely one of the fastest models I've run on local GPUs for its size, but only tried Nvidia cards. Edit: TIL it is MoE and only has 3.6B active, explains a lot.

Yeah, I'm probably wrong there. GPT OSS 20B is certainly much faster than some other models I've tried. I actually gave GPT OSS 20B a few prompts just now and it seems to respond as fast or faster than Qwen 3.5 9B. But I needed many more prompts for GPT OSS 20B to complete my contrived task, so progress felt much slower.

Re: Running local models on an M4 with 24GB memory

#56

Post by HDBaseT »

solenoid0937 wrote: post #41
quacker wrote: post #37

I could have used this article before I spent the weekend arriving to the same conclusion! Same laptop, and my contrived test was having it fix 50 or so lint errors in a small vibe-coded C++ repo. I wanted it to be able to handle a bunch of small tasks without getting stuck too often. GPT OSS 20B was usable but slow, and actually frequently made mistakes like adding or duplicating statements unnecessarily, listing th…

> It is absolutely not comparable to frontier models. This is not said often enough. Yes, local LLMs are great! But reading most HN posts on the subject, you'd think they're within reach of Opus 4.7. There is a very small, very vocal, very passionate crowd that dramatically overstates the capabilities of local LLMs on HN.

At least in my experience, local models are very far away from models like Opus 4.7 or ChatGPT 5.5 in coding and problem solving areas.

I find them useful in basic research and learning and question asking tasks. Although at the same time, a Wikipedia page read or a few Google searches likely could accomplish the same and has been able to for decades.

Re: Running local models on an M4 with 24GB memory

#57

Post by dempedempe »

BubbleRings wrote: post #46

People do use SOTA LLM’s for other things besides computer programming. For instance, if you are an independent inventor trying to write a patent while keeping your patent lawyer expenses to a minimum, you want to write as much of the first draft(s) of the patent as possible yourself. (You’ll save billable hours with your patent lawyer, and you’ll end up with a better patent because you’ll communicate your innovation…

It takes people years to learn how to write a good patent. If you gave your lawyer your attempt at writing your own patent, they might use the info to understand what you want (you're right about that), but a good lawyer would probably just start from scratch.

Imagine you're a contractor. You have a client who knows nothing about software development that wants you to write some software for them. They give you some code they generated with an LLM to get you started. Would you use the code or start over?

Re: Running local models on an M4 with 24GB memory

#58

Post by discordance »

soganess wrote: post #30

Getting so close to good! I consider Gemma 4 31B (dense / no MoE), the new baseline for local models. It's obviously worse than the frontier models, but it feels less like a science experiment than any previous local model I’ve run, including GPT OSS 120B and Nemotron Super 120B. On my M5 Max with 128 GB of RAM and the full 256K context window, I see RAM use spike to about 70 GB, with something like 14 GB of system o…

Could you please share your time to first token and tok/s?

Re: Running local models on an M4 with 24GB memory

#59

Post by gabagool »

tom_ wrote: post #48
guessmyname wrote: post #26

Earlier quoted context omitted.

A 128GiB MacBook Pro in Canada is what, north of CAD $11k after tax? That’s around USD $7k. At $20/month for a cloud AI subscription, you’re looking at almost 30 years of service for the same money. How long do people realistically expect a laptop to stay competitive with SOTA local models? Especially in a space where model sizes, context windows, and inference requirements keep moving every year. And even if the har…

But what if you were going to buy a laptop anyway? Obviously you can't do anything with less than 64 GBytes these days, so the question is just whether you go for the jump to 128. In the UK, it's currently an extra £800 to get a 128 GB vs the 64 GB equivalent. So that's more like 3 years of Claude - I think? - assuming current prices stay the same. Or: you might just feel like £800 isn't an unjustifiable amount of mo…

> Obviously you can't do anything with less than 64 GBytes these days

I don't think that's true. Plenty of people can run basic workflows at 8GB on the MacBook Neo and most others are fine at 16 GB.

Re: Running local models on an M4 with 24GB memory

#60

Post by canpan »

wktmeow wrote: post #49
canpan wrote: post #10

Recent models (Qwen 3.6 and Gemma) can really do coding locally. Feels like SOTA from maybe a year ago? But you would want about 32-40GB total memory. 24GB is just a bit short of that. A gaming PC with 16GB graphics card and 32GB RAM brings you very close to a usable coding system.

That’s the exact ram/vram combo of my desktop - what model would you suggest for that gaming pc setup?

I would recommend to start withQwen 3.6 35B at maybe Q5, it should be fast in that setup. For intelligence Qwen 3.7 27b, is smarter but will run much more slow. Others also mention gemma 4, which might be worth a try.
Post reply on HN