Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I ran Gemma 4 as a local model in Codex CLI

blog.danielvaughan.com

Post reply on HN
Print view View original

91–100 of 123 posts

Re: I ran Gemma 4 as a local model in Codex CLI

#91

Post by rsolva »

tuzemec wrote: post #9

I'm currently experimenting with running google/gemma-4-26b-a4b with lm studio ( https://lmstudio.ai/ ) and Opencode on a M3 Ultra with 48Gb RAM. And it seems to be working. I had to increase the context size to 65536 so the prompts from Opencode would work, but no other problems so far. I tried running the same on an M3 Max with less memory, but couldn't increase the context size enough to be useful with Opencode. I…

I run this model on my AMD RX7900XTX with 24GB VRAM with up to 4 concurrent chats and 512K context window in total. It is very fast (~100 t/s) and feels instant and very capable, and I have used Claude Code less and less these days.

Re: I ran Gemma 4 as a local model in Codex CLI

#92

Post by ozgrakkurt »

tuzemec wrote: post #9

I'm currently experimenting with running google/gemma-4-26b-a4b with lm studio ( https://lmstudio.ai/ ) and Opencode on a M3 Ultra with 48Gb RAM. And it seems to be working. I had to increase the context size to 65536 so the prompts from Opencode would work, but no other problems so far. I tried running the same on an M3 Max with less memory, but couldn't increase the context size enough to be useful with Opencode. I…

Not sure if you already tried but both GLM Flash and Qwen models are much better than Gemma for that in my experience.

I am using a 24GB GPU so it might be different in your case, but I doubt it.

Re: I ran Gemma 4 as a local model in Codex CLI

#93

Post by adrian_b »

cjbgkagh wrote: post #74

I've been playing with this for the last few days. The model is fast, pretty smart, and I am hitting the same tool use issues. This blog post is unusually pertinent. The model speed isn't an issue on my dual 4090s, the productivity is mainly limited by the intelligence (while high it's still not high enough for some tasks) and getting stuck in loops. What I would like is for it to be able detect when these things hap…

Google has replaced chat_template.jinja and tokenizer_config.json a few days ago in gemma-4-31B-it, which is supposed to have solved some problems related to tool invocation.

So if you have not updated your model, you should do it.

Re: I ran Gemma 4 as a local model in Codex CLI

#94

Post by minimaxir »

Confiks wrote: post #19
egorfine wrote: post #16

Related: I have upgraded my M4 Pro 24GB to M5 Pro 48GB yesterday. The same Gemma 4 MoE model (Q4) runs about 8x more t/s on M5 Pro and loads 2x times faster from disk to memory. Gonna run some more tests later today.

> The same Gemma 4 MoE model (Q4) As you have so much RAM I would suggest running Q8_0 directly. It's not slower (perhaps except for the initial model load), and might even be faster, while being almost identical in quality to the original model. And just to be sure: you're are running the MLX version, right? The mlx-community quantization seemed to be broken when I tried it last week (it spit out garbage), so I down…

Gemma 4 is not supported by the MLX engine yet.

Re: I ran Gemma 4 as a local model in Codex CLI

#95

Post by egorfine »

Confiks wrote: post #19
egorfine wrote: post #16

Related: I have upgraded my M4 Pro 24GB to M5 Pro 48GB yesterday. The same Gemma 4 MoE model (Q4) runs about 8x more t/s on M5 Pro and loads 2x times faster from disk to memory. Gonna run some more tests later today.

> The same Gemma 4 MoE model (Q4) As you have so much RAM I would suggest running Q8_0 directly. It's not slower (perhaps except for the initial model load), and might even be faster, while being almost identical in quality to the original model. And just to be sure: you're are running the MLX version, right? The mlx-community quantization seemed to be broken when I tried it last week (it spit out garbage), so I down…

> That too was broken in mlx-lm (it crashed), but has since been fixed on the main branch

Unfortunately I have got zero success running gemma with mlx-lm main branch. Can you point me out what is the right way? I have zero experience with mlx-lm.

Re: I ran Gemma 4 as a local model in Codex CLI

#96

Post by seemaze »

gertlabs wrote: post #84

Gemma 4 26B really is an outlier in its weight class. In our little known, difficult to game benchmarks, it scored about as well as GPT 5.2 and Gemini 3 Pro Preview on one-shot coding problems. It had me re-reviewing our entire benchmarking methodology. But it struggled in the other two sections of our benchmark: agentic coding and non-coding decision making. Tool use, iterative refinement, managing large contexts, a…

Thats funny, it failed my usual ‘hello world’ benchmark for LLM’s:

“Write a single file web page that implements a 1 dimensional bin fitting calculator using the best fit decreasing algorithm. Allow the user to input bin size, item size, and item quantity.”

Qwen3.5, Nematron, Step 3.5, gpt-oss all passed first go..

Re: I ran Gemma 4 as a local model in Codex CLI

#97

Post by iknowstuff »

gertlabs wrote: post #84

Gemma 4 26B really is an outlier in its weight class. In our little known, difficult to game benchmarks, it scored about as well as GPT 5.2 and Gemini 3 Pro Preview on one-shot coding problems. It had me re-reviewing our entire benchmarking methodology. But it struggled in the other two sections of our benchmark: agentic coding and non-coding decision making. Tool use, iterative refinement, managing large contexts, a…

Gemma 31B scoring below 26B-A4B?

Re: I ran Gemma 4 as a local model in Codex CLI

#98

Post by gertlabs »

iknowstuff wrote: post #97
gertlabs wrote: post #84

Gemma 4 26B really is an outlier in its weight class. In our little known, difficult to game benchmarks, it scored about as well as GPT 5.2 and Gemini 3 Pro Preview on one-shot coding problems. It had me re-reviewing our entire benchmarking methodology. But it struggled in the other two sections of our benchmark: agentic coding and non-coding decision making. Tool use, iterative refinement, managing large contexts, a…

Gemma 31B scoring below 26B-A4B?

In one shot coding, surprisingly, yes, by a decent amount. And it isn't a sample size issue. In agentic, no: https://gertlabs.com/?agentic=agentic

My early takeaway is that Gemma 26B-A4B is the best tuned out of the bunch, but being small and with few active params, it's severely constrained by context (large inputs and tasks with large required outputs tank Gemma 26B's performance). We're working on a clean visualization for this; the data is there.

It's not uncommon for a sub-release of a model to show improvements across the board on its model card, but actually have mixed real performance compared to its predecessor (sometimes even being worse on average).

Re: I ran Gemma 4 as a local model in Codex CLI

#100

Post by datadrivenangel »

gertlabs wrote: post #84

Gemma 4 26B really is an outlier in its weight class. In our little known, difficult to game benchmarks, it scored about as well as GPT 5.2 and Gemini 3 Pro Preview on one-shot coding problems. It had me re-reviewing our entire benchmarking methodology. But it struggled in the other two sections of our benchmark: agentic coding and non-coding decision making. Tool use, iterative refinement, managing large contexts, a…

Overall it's a very good open weights model! Notably I thought it makes more dumb coding mistakes than GPT-OSS on my M5, but it's fairly close overall.
Post reply on HN