I ran Gemma 4 as a local model in Codex CLI
blog.danielvaughan.com
I ran Gemma 4 as a local model in Codex CLI
1–10 of 123 posts
Re: I ran Gemma 4 as a local model in Codex CLI
#2Amazing. Thanks for your detailed posts on the bake-off between the Mac and GB10, Daniel, and on your learnings. I had trying similar on both compute platforms on my to-do list. Your post should save me a lot of debugs, sweat, and tears.
Re: I ran Gemma 4 as a local model in Codex CLI
#3I've been VERY impressed with Gemma4 (26B at the moment). It's the first time I've been able to use OpenCode via a llamacpp server reliably and actually get shit done.
In fact, I started using it as a coding partner while learning how to use the Godot game engine (and some custom 'skills' I pulled together from the official docs). I purposely avoided Claude and friends entirely, and just used Gemma4 locally this week... and it's really helped me figure out not just coding issues I was encountering, but also helped me sift through the documentation quite readily. I never felt like I needed to give in and use Claude.
Very, very pleased.
Re: I ran Gemma 4 as a local model in Codex CLI
#4With a nvidia spark or 128gb+ memory machine, you can get a good speed up on the 31B model if you use the 26B MoE as a draft model. It uses more memory but I’ve seen acceptance rate at around 70%+ using Q8 on both models
Re: I ran Gemma 4 as a local model in Codex CLI
#5This is genuinely very helpful. I'm planning a MacBook pro purchase with local inference in mind and now see I'll have to aim for a slightly higher memory option because the Gemma A4 26B MoE is not all that!
Re: I ran Gemma 4 as a local model in Codex CLI
#6Nothing about omlx?
Re: I ran Gemma 4 as a local model in Codex CLI
#7With a nvidia spark or 128gb+ memory machine, you can get a good speed up on the 31B model if you use the 26B MoE as a draft model. It uses more memory but I’ve seen acceptance rate at around 70%+ using Q8 on both models
1 token ahead or 2?
It's interesting - imo we'll soon have draft models specifically post-trained for denser, more complicated models. Wouldn't be surprised if diffusion models made a comeback for this - they can draft many tokens at once, and learning curves seem to top out at 90+% match for auto-regressive ones so quite interesting..
Re: I ran Gemma 4 as a local model in Codex CLI
#8Hey - I use the same, w/ both gemma4 and gpt-oss-*; some things I have to do for a good experience:
1) Pin to an earlier version of codex (sorry) - 0.55 is the best experience IME, but YMMV (see https://github.com/openai/codex/issues/11940, https://github.com/openai/codex/issues/8272).
2) Use the older completions endpoint (llama.cpp's responses support is incomplete - https://github.com/ggml-org/llama.cpp/issues/19138)
Re: I ran Gemma 4 as a local model in Codex CLI
#9I'm currently experimenting with running google/gemma-4-26b-a4b with lm studio (https://lmstudio.ai/) and Opencode on a M3 Ultra with 48Gb RAM.
And it seems to be working. I had to increase the context size to 65536 so the prompts from Opencode would work, but no other problems so far.
I tried running the same on an M3 Max with less memory, but couldn't increase the context size enough to be useful with Opencode.
It's also easy to integrate it with Zed via ACP. For now it's mostly simple code review tasks and generating small front-end related code snippets.
Re: I ran Gemma 4 as a local model in Codex CLI
#10For coding it makes no sense to use any quantization worse than Q6_K, from my experience. More quantized models make more mistakes and if for text processing it still can be fine, for coding it's not.