Well, but if I have a MacBook Air M4 with 16GB, I don't know what useful models can I run.
`brew install llmfit`
Running local models on an M4 with 24GB memory
101–110 of 189 posts
Re: Running local models on an M4 with 24GB memory
#102Getting so close to good! I consider Gemma 4 31B (dense / no MoE), the new baseline for local models. It's obviously worse than the frontier models, but it feels less like a science experiment than any previous local model I’ve run, including GPT OSS 120B and Nemotron Super 120B. On my M5 Max with 128 GB of RAM and the full 256K context window, I see RAM use spike to about 70 GB, with something like 14 GB of system o…
i use smaller model gemma e2b for most of my editing and it works surprisingly well. Workflow is planning with sota models and execution via small models. If you plan properly dont leave ambiguity for smaller model it works well.
Out of curiosity have you tried other small models? The e2b for me was unusable. Llama3.2 3b was better and that thing is a year old and I rarely use it now too.
Re: Running local models on an M4 with 24GB memory
#103Getting so close to good! I consider Gemma 4 31B (dense / no MoE), the new baseline for local models. It's obviously worse than the frontier models, but it feels less like a science experiment than any previous local model I’ve run, including GPT OSS 120B and Nemotron Super 120B. On my M5 Max with 128 GB of RAM and the full 256K context window, I see RAM use spike to about 70 GB, with something like 14 GB of system o…
Does gemma work better than qwen3 in your experience?
Not in mine. I see a lot of people talking about Gemma on here but in my circles pretty much everyone else is running qwen.
Re: Running local models on an M4 with 24GB memory
#104I could have used this article before I spent the weekend arriving to the same conclusion! Same laptop, and my contrived test was having it fix 50 or so lint errors in a small vibe-coded C++ repo. I wanted it to be able to handle a bunch of small tasks without getting stuck too often. GPT OSS 20B was usable but slow, and actually frequently made mistakes like adding or duplicating statements unnecessarily, listing th…
Try qwen3.6.35 a3b not qwen3.5 9b. It's completely different.
Re: Running local models on an M4 with 24GB memory
#105I could have used this article before I spent the weekend arriving to the same conclusion! Same laptop, and my contrived test was having it fix 50 or so lint errors in a small vibe-coded C++ repo. I wanted it to be able to handle a bunch of small tasks without getting stuck too often. GPT OSS 20B was usable but slow, and actually frequently made mistakes like adding or duplicating statements unnecessarily, listing th…
> It is absolutely not comparable to frontier models. This is not said often enough. Yes, local LLMs are great! But reading most HN posts on the subject, you'd think they're within reach of Opus 4.7. There is a very small, very vocal, very passionate crowd that dramatically overstates the capabilities of local LLMs on HN.
The guy is running potato models!
Re: Running local models on an M4 with 24GB memory
#106Earlier quoted context omitted.
Very different from my experience, Gemma 31b just solved a physics problem Opus 4.7 gave up on. I definitely don't think they're equivalent in general, Opus for sure is way smarter and way more likely to get things right on the edge, but it's still quite likely to get things wrong too it doesn't make it that useful for a lot of stuff. Conversely there are so many things that you would use an LLM for that they will bo…
Sure. Sample size = 1.
The models op is using are from a year ago. The big breakthroughs happened in April this past month
Re: Running local models on an M4 with 24GB memory
#107Earlier quoted context omitted.
> It is absolutely not comparable to frontier models. This is not said often enough. Yes, local LLMs are great! But reading most HN posts on the subject, you'd think they're within reach of Opus 4.7. There is a very small, very vocal, very passionate crowd that dramatically overstates the capabilities of local LLMs on HN.
This. I have seen way too many people who are overly optimistic about local LLMs. Having spent a decent amount of time playing with them on consumer nvidia GPUs, I understand well that they not going to be widely usable any time soon. Unfortunately not many people share that.
So the cofounder of hugging face made a post about qwen 3.6 being atclaude level of performance for the lols?
When were you trying local models? The model releases from April 2026 are a serious change in performance.
Re: Running local models on an M4 with 24GB memory
#108Earlier quoted context omitted.
> It is absolutely not comparable to frontier models. This is not said often enough. Yes, local LLMs are great! But reading most HN posts on the subject, you'd think they're within reach of Opus 4.7. There is a very small, very vocal, very passionate crowd that dramatically overstates the capabilities of local LLMs on HN.
This. I have seen way too many people who are overly optimistic about local LLMs. Having spent a decent amount of time playing with them on consumer nvidia GPUs, I understand well that they not going to be widely usable any time soon. Unfortunately not many people share that.
[dead]
Re: Running local models on an M4 with 24GB memory
#109Critics are (rightly) pointing to the fact that these models are not on par with SOTA for complex coding tasks. But many seems to forget that a large part of white collar office work is Excel crushing, file moving, translating dry legal documents, e-mail drafting, PPT drudgery, etc. These are absolutely doable with 30-35b+ models with the added benefit of keeping company data private.
I think the conclusion is flawed here? Sure qwen3.5 9b is nowhere near the sota models. It's 9b and was made a year ago? Everyone taking about local models is pumped about the models released in April this year. Qwen 3.6 27b and qwen 35b a3b if you have a sad GPU. Those are comparable to sota models, seriously.
Re: Running local models on an M4 with 24GB memory
#110[dead]