Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Running local models on an M4 with 24GB memory

jola.dev

Post reply on HN
Print view View original

11–20 of 189 posts

Re: Running local models on an M4 with 24GB memory

#11

Post by sourc3 »

I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment.

As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to openrouter and run the largest qwen model for < $2-3/day with serious use for personal projects.

Re: Running local models on an M4 with 24GB memory

#12

Post by DrBenCarson »

canpan wrote: post #10

Recent models (Qwen 3.6 and Gemma) can really do coding locally. Feels like SOTA from maybe a year ago? But you would want about 32-40GB total memory. 24GB is just a bit short of that. A gaming PC with 16GB graphics card and 32GB RAM brings you very close to a usable coding system.

How are you using that RAM with the GPU?

Re: Running local models on an M4 with 24GB memory

#13

Post by hparadiz »

sourc3 wrote: post #11

I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment. As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to…

How does it (the openrouter version) compare to ChatGPT 5.5 or Claude Opus 4.6?

Re: Running local models on an M4 with 24GB memory

#15

Post by canpan »

DrBenCarson wrote: post #12
canpan wrote: post #10

Recent models (Qwen 3.6 and Gemma) can really do coding locally. Feels like SOTA from maybe a year ago? But you would want about 32-40GB total memory. 24GB is just a bit short of that. A gaming PC with 16GB graphics card and 32GB RAM brings you very close to a usable coding system.

How are you using that RAM with the GPU?

Llama.cpp with automatic offload to main memory. You can also use Ollama, it is easier, but slower.

Re: Running local models on an M4 with 24GB memory

#16

Post by carbocation »

sourc3 wrote: post #11

I am running qwen 3.6 9b quantized model on my m4 pro 48gb and it is barely useful to do some basic pi.dev/cc driven development. I think 128gb desktops are the sweet setup to actually get meaningful work done. However, getting your hands on one of these machines is difficult at the moment. As much fun as it is to run these things locally don’t forget that your time is not free. I am slowly migrating my use cases to…

Was the choice of such a small model driven by a desire for high tok/sec? I ask because an m4 pro 48gb machine can run larger models (if model intelligence is the thing that would make it more useful).

Re: Running local models on an M4 with 24GB memory

#20

Post by ai_fry_ur_brain »

canpan wrote: post #10

Recent models (Qwen 3.6 and Gemma) can really do coding locally. Feels like SOTA from maybe a year ago? But you would want about 32-40GB total memory. 24GB is just a bit short of that. A gaming PC with 16GB graphics card and 32GB RAM brings you very close to a usable coding system.

[flagged]
Post reply on HN