Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I ran Gemma 4 as a local model in Codex CLI

blog.danielvaughan.com

Post reply on HN
Print view View original

111–120 of 123 posts

Re: I ran Gemma 4 as a local model in Codex CLI

#111

Post by neonstatic »

gertlabs wrote: post #84

Gemma 4 26B really is an outlier in its weight class. In our little known, difficult to game benchmarks, it scored about as well as GPT 5.2 and Gemini 3 Pro Preview on one-shot coding problems. It had me re-reviewing our entire benchmarking methodology. But it struggled in the other two sections of our benchmark: agentic coding and non-coding decision making. Tool use, iterative refinement, managing large contexts, a…

I have very mixed feelings about that model. I want to like it. It's very fast and seems to be fit for many uses. I strongly dislike its "personality", but it responds well to system prompts.

Unfortunately, my experience with it as a coding assistant is very poor. It doesn't understand libraries it seems to know about, it doesn't see root causes of problems I want it to solve, and it refuses to use MCP tools even when asked. It has a very strong fixation on the concept of time. Anything past January 2025, which I think is its knowledge cutoff, the model will label as "science fiction" or "their fantasy" and role play from there.

Re: I ran Gemma 4 as a local model in Codex CLI

#113

Post by adrian_b »

gertlabs wrote: post #98
iknowstuff wrote: post #97

Earlier quoted context omitted.

Gemma 31B scoring below 26B-A4B?

In one shot coding, surprisingly, yes, by a decent amount. And it isn't a sample size issue. In agentic, no: https://gertlabs.com/?agentic=agentic My early takeaway is that Gemma 26B-A4B is the best tuned out of the bunch, but being small and with few active params, it's severely constrained by context (large inputs and tasks with large required outputs tank Gemma 26B's performance). We're working on a clean visualiz…

In early tests the performance of gemma-4-31B was affected by tokenizer bugs in many of the existing backends, like llama.cpp, which were later corrected by their maintainers.

Moreover, tool invocation had problems that were later corrected by Google in an updated chat template.

So any early benchmarks that have shown the dense model as inferior to the MoE model are likely to be flawed and they must be repeated after updating both the inference backend and the model.

All benchmarks that I have seen after the bugs were fixed have shown the dense model as clearly superior in quality, even if much slower.

Re: I ran Gemma 4 as a local model in Codex CLI

#114

Post by gertlabs »

adrian_b wrote: post #113
gertlabs wrote: post #98

Earlier quoted context omitted.

In one shot coding, surprisingly, yes, by a decent amount. And it isn't a sample size issue. In agentic, no: https://gertlabs.com/?agentic=agentic My early takeaway is that Gemma 26B-A4B is the best tuned out of the bunch, but being small and with few active params, it's severely constrained by context (large inputs and tasks with large required outputs tank Gemma 26B's performance). We're working on a clean visualiz…

In early tests the performance of gemma-4-31B was affected by tokenizer bugs in many of the existing backends, like llama.cpp, which were later corrected by their maintainers. Moreover, tool invocation had problems that were later corrected by Google in an updated chat template. So any early benchmarks that have shown the dense model as inferior to the MoE model are likely to be flawed and they must be repeated after…

We add samples every week, so I'm curious if the numbers will move.

They did a similar re-release during the Gemini 3.1 Pro Preview rollout, and released a custom-tools version with its own slug, which performs MUCH better on custom harnesses (mostly because the original release could not figure out tool call formatting at all).

Re: I ran Gemma 4 as a local model in Codex CLI

#115

Post by tgv »

OutOfHere wrote: post #28
tgv wrote: post #22

Earlier quoted context omitted.

I don't quite get why you feel so strongly about it that this should be a deal breaker for everyone. It's really much better than a wrong answer, for everyone.

> It's really much better than a wrong answer That is a bad premise and a false dichotomy, because most medical questions are simple, with well-known standard answers. ChatGPT and Gemini answer such questions correctly, also finding glaring omissions by doctors, even without having to look up information. As for the medical questions that are not simple, the ones that require looking up information, the model should…

Odd, but for future discussions: it's nor a premise, nor a dichotomy, but a judgement. A premise would be something like: bad medical advice is bad. A dichotomy would be: Gemma can offer no medical advice or bad medical advice. It then remains to be seen if they are bad or false.

But Gemma is a "small" model, and may not be expected to answer all questions. Medical questions are particularly sensitive, so it's quite possible they decided to err on the side of caution and plausible deniability. That doesn't rule out the model has other virtues.

Re: I ran Gemma 4 as a local model in Codex CLI

#116

Post by prettyblocks »

gertlabs wrote: post #84

Gemma 4 26B really is an outlier in its weight class. In our little known, difficult to game benchmarks, it scored about as well as GPT 5.2 and Gemini 3 Pro Preview on one-shot coding problems. It had me re-reviewing our entire benchmarking methodology. But it struggled in the other two sections of our benchmark: agentic coding and non-coding decision making. Tool use, iterative refinement, managing large contexts, a…

For me the vision/OCR is much better than other models in weights class.

Re: I ran Gemma 4 as a local model in Codex CLI

#117

Post by stratos123 »

mft_ wrote: post #25
OutOfHere wrote: post #21

Gemma 4 is a strongly censored model, so much so that it refused to answer medical and health related questions, even basic ones. No one should be using it, and if this is the best that Google can do, it should stop now. Other models do not have such ridiculous self-imposed problems.

I suspect a possible future of local models is extreme specialisation - you load a Python-expert model for Python coding, do your shopping with a model focused just on this task, have a model specialised in speech-to-text plus automation to run your smart home, and so on. This makes sense: running a huge model for a task that only uses a small fraction of its ability is wasteful, and home hardware especially isn't su…

> I suspect a possible future of local models is extreme specialisation - you load a Python-expert model for Python coding, do your shopping with a model focused just on this task, have a model specialised in speech-to-text plus automation to run your smart home, and so on.

I'd find this very surprising, since a lot of cognitive skills are general. At least on the scale of "being trained on a lot of non-Python code improves a model's capabilities in Python", but maybe even "being trained on a lot of unrelated tasks that require perseverance improves a model's capabilities in agentic coding".

For this reason there are currently very few specialist models - training on specialized datasets just doesn't work all that well. For example, there are the tiny Jetbrains Mellum models meant for in-editor autocomplete, but even those are AFAIK merely fine-tuned on specific languages, while their pretraining dataset is mixed-language.

Re: I ran Gemma 4 as a local model in Codex CLI

#118

Post by qingcharles »

hak8or wrote: post #108
qingcharles wrote: post #102

Earlier quoted context omitted.

I spun up a GPU on Runpod and tried the 31b full res and it was really impressive. I'm now using it via the Google API which gives you 1500 requests a day for free, IIRC.

Be very careful about using googles apis as a consumer, they have poor rate limiting and ineffective anomoly protection. I (a hobbyist running a small side project for a dollar or two a month in normal usage, so my account is marked as "individual") got hit with a ~$17,000 bill from Google cloud because some combination of key got leaked or my homelab got compromised, and the attacker consumed tens of thousands in ge…

Agree totally. I'm super paranoid and anxious about this issue. I've seen too many horror stories posted on Reddit. I did set alarms at $10 a day on the account, but those are only alarms and it could be thousands over before I see them.

I think Google did finally implement hard limits this month and I need to go and find that setting, but it's useless if, like you say, they have shitty rate limiting and measurement so that you're way over the limit before they stop you.

Re: I ran Gemma 4 as a local model in Codex CLI

#119

Post by egorfine »

Confiks wrote: post #110
egorfine wrote: post #95

Earlier quoted context omitted.

> That too was broken in mlx-lm (it crashed), but has since been fixed on the main branch Unfortunately I have got zero success running gemma with mlx-lm main branch. Can you point me out what is the right way? I have zero experience with mlx-lm.

Get into a venv, and run: > pip3 install git+ https://github.com/ml-explore/mlx-lm.git > ./venv/bin/mlx_lm.generate --model "$MODEL" --temp 1.0 --top-p 0.95 --top-k 64 --max-tokens 128000 --prompt "Hello world" Where $MODEL is an unsloth model like: - unsloth/gemma-4-E4B-it-UD-MLX-4bit - unsloth/gemma-4-26b-a4b-it-UD-MLX-4bit

Thanks!
Post reply on HN