I ran Gemma 4 as a local model in Codex CLI • Hacker News Board

Re: I ran Gemma 4 as a local model in Codex CLI

#121

Post by theshrike79 » Wed, Apr 15, 2026, 11:49 AM UTC

usagisushi wrote: post #69

tuzemec wrote: post #9

I'm currently experimenting with running google/gemma-4-26b-a4b with lm studio ( https://lmstudio.ai/ ) and Opencode on a M3 Ultra with 48Gb RAM. And it seems to be working. I had to increase the context size to 65536 so the prompts from Opencode would work, but no other problems so far. I tried running the same on an M3 Max with less memory, but couldn't increase the context size enough to be useful with Opencode. I…

I have a similar setup. It might be worth checking out pi-coding-agent [0]. The system prompt and tools have very little overhead ( [0] https://www.npmjs.com/package/@mariozechner/pi-coding-agent#...

Pi is _really_ good for personal stuff, but since it lacks every single safety imaginable, it's not realy something one can deploy in a corporate environment :D

Re: I ran Gemma 4 as a local model in Codex CLI

#122

Post by shaz0x » Wed, Apr 15, 2026, 10:28 PM UTC

zihotki wrote: post #10

For coding it makes no sense to use any quantization worse than Q6_K, from my experience. More quantized models make more mistakes and if for text processing it still can be fine, for coding it's not.

On mobile the Q4 vs Q6 tradeoff flips. Gemma 4 E2B at Q4_K_M barely fits in RAM on a 6GB Android, so Q6 isn't on the table. In practice the Q4 hit shows up in tool-call reliability more than general reasoning, which is usually fine for a constrained skill surface.

Re: I ran Gemma 4 as a local model in Codex CLI

#123

Post by magic_hamster » Fri, Apr 17, 2026, 5:41 AM UTC

fortyseven wrote: post #60

magic_hamster wrote: post #31

Ollama is the worst engine you could use for this. Since you are already running on an Nvidia stack for the dense model, you should serve this with vLLM. With 128GB you could try for the original safetensors even though you might need to be careful with caches and context length.

Strangely, I haven't had a lot of luck with vLLM; I finally ended up ditching Ollama and going straight to the tap with llama-serve in llamacpp. No regrets.

Good job. llama.cpp is already much better.