Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models on an M4 with 24GB memory

jola.dev

Post reply on HN
Print view View original

141–150 of 189 posts

Re: Running local models on an M4 with 24GB memory

#141

Post by threetonesun »

Xeoncross wrote: post #136

Is it better to have an M4-M5 Pro with 32GB of ram or an M1-M2 Max with 64GB of ram? They seem about the same price. It seems like cache layers like https://omlx.ai make more RAM better than more GPU cores or faster CPUs cores, but I'm curious if someone has tested both.

When I was considering a local setup the M1 Ultra Studios with 128 GB of RAM seemed to be the best price:performance at the time. I think RAM always wins out.

Also minor note: the M4/5 Pros come in multiples of 12, so it's a 24/36 or 48GB set up.

Re: Running local models on an M4 with 24GB memory

#142

Post by anon373839 »

solenoid0937 wrote: post #140
2ndorderthought wrote: post #107

Earlier quoted context omitted.

So the cofounder of hugging face made a post about qwen 3.6 being atclaude level of performance for the lols? When were you trying local models? The model releases from April 2026 are a serious change in performance.

It's just not there yet. I have tried all the models from April, including the Gemma 4 variants. These are so far from Opus it's not even funny. They are not close to being in the same league. Gemma might be like a frontier model from a couple years ago, but with much worse performance in long context chats.

Hm. I think there is a bit of a shifting goalpost dynamic at play here. Those April releases, even the fast MoE versions, are better than big cloud models from 18 months ago. I remember when everyone was gushing about Sonnet 3.7 and what a transformative experience development was using it. So was it useful or wasn’t it? A tool doesn’t lose its usability just because a better one comes along.

To me, these small local LLMs are highly useful (and this “usable”) even though they don’t match the output of today’s frontier models.

Re: Running local models on an M4 with 24GB memory

#143

Post by spwa4 »

gertlabs wrote: post #85
soganess wrote: post #30

Getting so close to good! I consider Gemma 4 31B (dense / no MoE), the new baseline for local models. It's obviously worse than the frontier models, but it feels less like a science experiment than any previous local model I’ve run, including GPT OSS 120B and Nemotron Super 120B. On my M5 Max with 128 GB of RAM and the full 256K context window, I see RAM use spike to about 70 GB, with something like 14 GB of system o…

The small Qwen 3.6 models handle context a little better than Gemma 4, but Gemma 4 26B in particular has such small and efficient solutions which are really smart for its weight class. I was so impressed with its performance in our benchmark upon release that I wrote a blog post about it [0], although its position on the leaderboard later fell a bit as we ran it in more long context agentic coding environments. [0] h…

Here's a great explanation why:

https://www.youtube.com/watch?v=_A367W_qvc8

Google's messing with the context. LOTS of speed for a little worse long-context performance.

Re: Running local models on an M4 with 24GB memory

#144

Post by 2ndorderthought »

solenoid0937 wrote: post #140
2ndorderthought wrote: post #107

Earlier quoted context omitted.

So the cofounder of hugging face made a post about qwen 3.6 being atclaude level of performance for the lols? When were you trying local models? The model releases from April 2026 are a serious change in performance.

It's just not there yet. I have tried all the models from April, including the Gemma 4 variants. These are so far from Opus it's not even funny. They are not close to being in the same league. Gemma might be like a frontier model from a couple years ago, but with much worse performance in long context chats.

Correct they aren't opus. They are sonnet with a little hand holding. They also run on a single GPU at 40 tps.

No one is saying a local model will give you anthropics business in a 5min download. People are saying, "hmm, maybe I should do this one locally". People are also saying "this is surprisingly good enough for me given the trade offs"

Re: Running local models on an M4 with 24GB memory

#145

Post by 2ndorderthought »

anon373839 wrote: post #142
solenoid0937 wrote: post #140

Earlier quoted context omitted.

It's just not there yet. I have tried all the models from April, including the Gemma 4 variants. These are so far from Opus it's not even funny. They are not close to being in the same league. Gemma might be like a frontier model from a couple years ago, but with much worse performance in long context chats.

Hm. I think there is a bit of a shifting goalpost dynamic at play here. Those April releases, even the fast MoE versions, are better than big cloud models from 18 months ago. I remember when everyone was gushing about Sonnet 3.7 and what a transformative experience development was using it. So was it useful or wasn’t it? A tool doesn’t lose its usability just because a better one comes along. To me, these small local…

Completely agree. I would even shift the 18months up a bit. I have been impressed with qwen3.6

Re: Running local models on an M4 with 24GB memory

#146

Post by hugmynutus »

ChrisMarshallNY wrote: post #139

> The longer you let it drive without constraints, the worse the wreckage gets. The velocity makes you think you're winning right up until the moment everything collapses simultaneously. In my experience (so far), I can’t let the LLM write too much in one go. I need to test the hell out of what it gives me, and I can’t ask for too much, at one time. I tend to ask it to “flesh out” functions, where I have a signature,…

> What it gives me in Swift, most closely resembles stuff that enthusiastic newer folks would do, and want to show off.

The same is true for rust-lang. Code that will immediately clone/re-allocate anything passed by reference and collect everything to the heap that is passed by `Iterator`/`IntoIterator`.

It is a massive performance anti-pattern and the hallmark of somebody "struggling" with the borrow checker. Naturally a lot of 1st & 2nd 'I just learned rust' projects lean on it. Which is totally fine for humans, you're learning. But with LLMs that pattern is now burned into their eigenvectors with the heat of a billion hours of H100 training time.

It has gotten to a point that all code I generate with Opus or Codex if there as iterator or reference in the argument, I start a fresh context, with a sort of `remove unnecessary clones, collections, and copies from the following code: {{code}}`

Re: Running local models on an M4 with 24GB memory

#147

Post by claysmithr »

This is very good, local models run well on my M3 Air 24GB, to the point where I may prefer it even if it takes longer. The benefits are

  - private

  - local

  - no internet required

  - works well enough for most tasks

  - "free"

  - will pop this "AI" bubble as word spreads
I got pretty good results with the model in the article on my machine. Sure, it took forever, but that doesn't matter to me as much, and it's kind of cool just watching it do its thing through LM studio. The result was also impressive enough for me that I would actually use it.

Why pay $20/mo when local is good enough?

Re: Running local models on an M4 with 24GB memory

#148

Post by Casteil »

sourc3 wrote: post #11

I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment. As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to…

Why not 35b-a3b? ...or gemma4:26b-a4b? Both will be more capable than 9b and run at roughly similar (perhaps faster) speeds

Re: Running local models on an M4 with 24GB memory

#149

Post by Casteil »

noashavit wrote: post #134

Gemma4 is a huge improvement and it's fast. Qwen 3.5 really slows down my machine though. LMK if there is a better model to use for the code assist aspect- performance wise

Qwen3.5/3.6 are really prone to looping and 'overthinking'. Gemma4 doesn't seem to have the same problems.

Re: Running local models on an M4 with 24GB memory

#150

Post by alfiedotwtf »

soganess wrote: post #30

Getting so close to good! I consider Gemma 4 31B (dense / no MoE), the new baseline for local models. It's obviously worse than the frontier models, but it feels less like a science experiment than any previous local model I’ve run, including GPT OSS 120B and Nemotron Super 120B. On my M5 Max with 128 GB of RAM and the full 256K context window, I see RAM use spike to about 70 GB, with something like 14 GB of system o…

What's your opinion with Gemma 4 vs Qwen3.6?
Post reply on HN