Running local models on an M4 with 24GB memory • Hacker News Board

Re: Running local models on an M4 with 24GB memory

#21

Post by nl » Mon, May 11, 2026, 12:19 AM UTC

I think it's useful to be realistic about what you can do with a local model, especially something as small as the 9B the author is using. A 9B model is around the level of Sonnet 3.6 - it can do autocomplete and small functions but it loses track trying to understand large problems.

But the are interesting and fun to play with! I do a LOT of work on local agent harnesses etc, mostly for fun.

My current project is a zero install agent: https://gemma-agent-explainer.nicklothian.com/ - Python, SQL and React all run completely in browser. Gemma E4B is recommended for the best experience!

This is under heavy development, needs Chrome for both HTML5 Filesystem API support and LiteRT (although most Chromium based browsers can be made to work with it)

It's different to most agents because it is zero install: the model runs in the browser using LiteRT/LiteLLM (which gives better performance than Transformers.js), and Filesystem API gives it optional sandbox access to a directory to read from.

It is self documenting - you can ask questions like "How is the system prompt used" in the live help pane and it has access to its own source code.

There's quite a lot there: press "Tour" to see it all.

Will be open source next week.

Re: Running local models on an M4 with 24GB memory

#22

Post by elij » Mon, May 11, 2026, 12:20 AM UTC

sourc3 wrote: post #11

I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment. As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to…

I'm using the 30b MOE model on same spec with 65k tokens as a sub agent with tooling and it absolutely writes decent code. The dense 9b I agree wasn't great.

Re: Running local models on an M4 with 24GB memory

#23

Post by ai_fry_ur_brain » Mon, May 11, 2026, 12:21 AM UTC

nl wrote: post #21

I think it's useful to be realistic about what you can do with a local model, especially something as small as the 9B the author is using. A 9B model is around the level of Sonnet 3.6 - it can do autocomplete and small functions but it loses track trying to understand large problems. But the are interesting and fun to play with! I do a LOT of work on local agent harnesses etc, mostly for fun. My current project is a…

[flagged]

Re: Running local models on an M4 with 24GB memory

#24

Post by sourc3 » Mon, May 11, 2026, 12:23 AM UTC

carbocation wrote: post #16

sourc3 wrote: post #11

I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment. As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to…

Was the choice of such a small model driven by a desire for high tok/sec? I ask because an m4 pro 48gb machine can run larger models (if model intelligence is the thing that would make it more useful).

Yes that was my goal. Also noticed a huge performance gain going from ollama to mlx. Your mileage may vary.

Re: Running local models on an M4 with 24GB memory

#25

Post by sourc3 » Mon, May 11, 2026, 12:26 AM UTC

hparadiz wrote: post #13

sourc3 wrote: post #11

I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment. As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to…

How does it (the openrouter version) compare to ChatGPT 5.5 or Claude Opus 4.6?

Good enough. It gets 60-70% of the work I need done for a lot less $ (keep in mind I am using these for personal projects that doesn’t generate revenue). If I was using it with the hopes of making money I think I would just use Codex at this point.

Re: Running local models on an M4 with 24GB memory

#26

Post by guessmyname » Mon, May 11, 2026, 12:29 AM UTC

nu11ptr wrote: post #17

Still trying to understand if a Macbook Pro M5 Max with 128GB is likely going to be able to run coding models well enough that I can cancel my Codex, or even go down to the $20/month plan.

A 128GiB MacBook Pro in Canada is what, north of CAD $11k after tax? That’s around USD $7k. At $20/month for a cloud AI subscription, you’re looking at almost 30 years of service for the same money.

How long do people realistically expect a laptop to stay competitive with SOTA local models? Especially in a space where model sizes, context windows, and inference requirements keep moving every year.

And even if the hardware lasts, the local experience usually doesn’t. A heavily quantized local model running at tolerable speeds on consumer hardware is still nowhere near frontier hosted models in reasoning, coding, multimodal capability, tool use, or reliability.

The economics just don’t make sense to me unless you specifically need offline inference, privacy guarantees, or low latency for a niche workflow. Otherwise you’re tying up $10k upfront to run an approximation of what you can already access through a subscription that continuously improves over time.

You could literally put the difference into index funds and probably cover the subscription indefinitely from the returns alone, even accounting for gradual price increases.

Re: Running local models on an M4 with 24GB memory

#27

Post by sjones671 » Mon, May 11, 2026, 12:29 AM UTC

sourc3 wrote: post #11

I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment. As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to…

Thanks for saying this. There's so much nonsense out there online about local models being better than Opus 4.7 and the like. It's just not true for regular users.

I have a brand new M5 MacBook Pro - top end with all the specs and I've tried local models and they're barely functional.

Re: Running local models on an M4 with 24GB memory

#28

Post by nl » Mon, May 11, 2026, 12:35 AM UTC

ai_fry_ur_brain wrote: post #23

nl wrote: post #21

I think it's useful to be realistic about what you can do with a local model, especially something as small as the 9B the author is using. A 9B model is around the level of Sonnet 3.6 - it can do autocomplete and small functions but it loses track trying to understand large problems. But the are interesting and fun to play with! I do a LOT of work on local agent harnesses etc, mostly for fun. My current project is a…

[flagged]

I think knowledge is power.

I think that the more people who try local models (especially the larger ones) the better.

I sometimes get the impression that many people claiming that local models are as good as frontier models work in "token poor" environments. If you can't build large-scale programs using at least Opus 4.5+ then it's difficult to compare. They compare something like Qwen 27B with Sonnet and see that it is nearly as good, but miss that the frontier models are a lot better.

That knowledge is power, too.

I personally can help making local models more accessible. I can't make Opus cheaper.

Re: Running local models on an M4 with 24GB memory

#29

Post by [deleted] » Mon, May 11, 2026, 12:35 AM UTC

sourc3 wrote: post #11

I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment. As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to…

[deleted]

Re: Running local models on an M4 with 24GB memory

#30

Post by soganess » Mon, May 11, 2026, 12:36 AM UTC

Getting so close to good!

I consider Gemma 4 31B (dense / no MoE), the new baseline for local models. It's obviously worse than the frontier models, but it feels less like a science experiment than any previous local model I’ve run, including GPT OSS 120B and Nemotron Super 120B.

On my M5 Max with 128 GB of RAM and the full 256K context window, I see RAM use spike to about 70 GB, with something like 14 GB of system overhead. A 64 GB Panther Lake machine with the full Arc B390, or a 48 GB Snapdragon X2 Elite machine, could probably run it with a 128K to 256K context window. Maybe you can squeeze it into 32GB (27.5GB usable) with a 32K context window?

Even last year, seeing this kinda performance on a mainstream-ish/plus configuration would have seemed like a pipe dream.