Google releases Gemma 4 open models • Hacker News Board

Re: Google releases Gemma 4 open models

#191

Post by lysace » Thu, Apr 02, 2026, 7:24 PM UTC

wordpad wrote: post #83

simonw wrote: post #77

I ran these in LM Studio and got unrecognizable pelicans out of the 2B and 4B models and an outstanding pelican out of the 26b-a4b model - I think the best I've seen from a model that runs on my laptop. https://simonwillison.net/2026/Apr/2/gemma-4/ The gemma-4-31b model is completely broken for me - it just spits out "---\n" no matter what prompt I feed it. I got a pelican out of it via the AI Studio API hosted model…

Do you think it's just part of their training set now?

Seems very likely, even if Google has behaved ethically.

Simon and YC/HN has published/boosted these gradual improvements and evaluations for quite some time now.

There is a https://simonwillison.net/robots.txt but it allows pretty much everything, AI-wise.

Re: Google releases Gemma 4 open models

#192

Post by sdenton4 » Thu, Apr 02, 2026, 7:28 PM UTC

osti wrote: post #185

WarmWash wrote: post #49

Earlier quoted context omitted.

I am unable to shake that the Chinese models all perform awfully on the private arc-agi 2 tests.

But is arc-agi really that useful though? Nowadays it seems to me that it's just another benchmark that needs to be specifically trained for. Maybe the Chinese models just didn't focus on it as much.

Doing great on public datasets and underperforming on private benchmarks is not a good look.

Re: Google releases Gemma 4 open models

#193

Post by hikarudo » Thu, Apr 02, 2026, 7:29 PM UTC

Also checkout Deepmind's "The Gemma 4 Good Hackathon" on kaggle:

https://www.kaggle.com/competitions/gemma-4-good-hackathon

Re: Google releases Gemma 4 open models

#194

Post by nabakin » Thu, Apr 02, 2026, 7:30 PM UTC

moffkalast wrote: post #127

nabakin wrote: post #114

Earlier quoted context omitted.

It's easy to game and human evaluation data has its trade-offs, but it's way easier to fake public benchmark results. I wish we had a source of high quality private benchmark results across a vast number of models like Lmarena. Having high quality human evaluation data would be a plus too.

Well there was this one [0] which is a black box but hasn't really been kept up to date with newer releases. Arguably we'd need lots of these since each one could be biased towards some use case or sell its test set to someone with more VC money than sense. [0] https://oobabooga.github.io/benchmark.html

I know Arc AGI 2 has a private test set and they have a good amount of results[0] but it's not a conventional benchmark.

Looking around, SWE Rebench seems to have decent protection against training data leaks[1]. Kagi has one that is fully private[2]. One on HuggingFace that claims to be fully private[3]. SimpleBench[4]. HLE has a private test set apparently[5]. LiveBench[6]. Scale has some private benchmarks but not a lot of models tested[7]. vals.ai[8]. FrontierMath[9]. Terminal Bench Pro[10]. AA-Omniscience[11].

So I guess we do have some decent private benchmarks out there.

[0] https://arcprize.org/leaderboard

[1] https://swe-rebench.com/about

[2] https://help.kagi.com/kagi/ai/llm-benchmark.html

[3] https://huggingface.co/spaces/DontPlanToEnd/UGI-Leaderboard

[4] https://simple-bench.com/

[5] https://agi.safe.ai/

[6] https://livebench.ai/

[7] https://labs.scale.com/leaderboard

[8] https://www.vals.ai/about

[9] https://epoch.ai/frontiermath/

[10] https://github.com/alibaba/terminal-bench-pro

[11] https://artificialanalysis.ai/articles/aa-omniscience-knowle...

Re: Google releases Gemma 4 open models

#195

Post by vessenes » Thu, Apr 02, 2026, 7:31 PM UTC

zozbot234 wrote: post #157

vessenes wrote: post #141

Earlier quoted context omitted.

I'll pipe in - a series of Mac optimized MOEs which can stream experts just in time would be really amazing. And popular; I'm guessing in the next year we'll be able to run a very able openclaw with a stack like that. You'll get a lot of installs there. If I were a PM at Gemma, I'd release a stack for each Mac mini memory size.

Expert streaming is something that has to be implemented by the inference engine/library, the model architecture itself has very little to do with it. It's a great idea (for local inference; it uses too much power at scale), but making it work really well is actually not that easy. (I've mentioned this before but AIUI it would require some new feature definitions in GGUF, to allow for coalescing model data about any…

I’ve been doing some low-key testing on smaller models, and it looks to me like it’s possible to train an MOE model with characteristics that are helpful for streaming… For instance, you could add a loss function to penalize expert swapping both in a single forward, pass and across multiple forward passes. So I believe there is a place for thinking about this on the model training side.

Re: Google releases Gemma 4 open models

#196

Post by nabakin » Thu, Apr 02, 2026, 7:34 PM UTC

melodyogonna wrote: post #181

nabakin wrote: post #98

Earlier quoted context omitted.

Faster than TensorRT-LLM on Blackwell? Or do you not consider TensorRT-LLM open source because some dependencies are closed source?

I reviewed the TensorRT-LLM commit history from the past few days and couldn't find any updates regarding Gemma 4 support. By contrast, here is the reference for MAX: https://github.com/modular/modular/commit/57728b23befed8f3b4...

If OP meant they have the fastest implementation of Gemma 4 on Blackwell at the moment, I guess that is technically true. I doubt that will hold up when TensorRT-LLM finishes their implementation though.

Re: Google releases Gemma 4 open models

#197

Post by coder68 » Thu, Apr 02, 2026, 7:36 PM UTC

kcb wrote: post #165

coder68 wrote: post #159

Earlier quoted context omitted.

120B would be great to have if you have it stashed away somewhere. GPT-OSS-120B still stands as one of the best (and fastest) open-weights models out there. A direct competitor in the same size range would be awesome. The closest recent release was Qwen3.5-122B-A10B.

Nemotron 3 Super was released recently. That's a direct competitor to gpt-oss-120b. https://developer.nvidia.com/blog/introducing-nemotron-3-sup...

I gave it a whirl but was unenthused. I'll try it again, but so far have not really enjoyed any of the nvidia models, though they are best in class for execution speed.

Re: Google releases Gemma 4 open models

#198

Post by hrmtst93837 » Thu, Apr 02, 2026, 7:37 PM UTC

jeffbee wrote: post #27

jwr wrote: post #3

Really looking forward to testing and benchmarking this on my spam filtering benchmark. gemma-3-27b was a really strong model, surpassed later by gpt-oss:20b (which was also much faster). qwen models always had more variance.

Does spam filtering really need a better model? My impression is that the whole game is based on having the best and freshest user-contributed labels.

[flagged]

Re: Google releases Gemma 4 open models

#199

Post by swalsh » Thu, Apr 02, 2026, 7:38 PM UTC

I gave the same prompt (a small rust project that's not easy, but not overly sophisticated) to both Gemma-4 26b and Qwen 3.5 27b via OpenCode. Qwen 3.5 ran for a bit over an hour before I killed it, Gemma 4 ran for about 20 minutes before it gave up. Lots of failed tool calls.

I asked codex to write a summary about both code bases.

"Dev 1" Qwen 3.5

"Dev 2" Gemma 4

Dev 1 is the stronger engineer overall. They showed better architectural judgment, stronger completeness, and better maintainability instincts. The weakness is execution rigor: they built more, but didn’t verify enough, so important parts don’t actually hold up cleanly.

Dev 2 looks more like an early-stage prototyper. The strength is speed to a rough first pass, but the implementation is much less complete, less polished, and less dependable. The main weakness is lack of finish and technical rigor.

If I were choosing between them as developers, I’d take Dev 1 without much hesitation.

Looking at the code myself, i'd agree with codex.

Re: Google releases Gemma 4 open models

#200

Post by baal80spam » Thu, Apr 02, 2026, 7:42 PM UTC

culi wrote: post #171

simonw wrote: post #77

I ran these in LM Studio and got unrecognizable pelicans out of the 2B and 4B models and an outstanding pelican out of the 26b-a4b model - I think the best I've seen from a model that runs on my laptop. https://simonwillison.net/2026/Apr/2/gemma-4/ The gemma-4-31b model is completely broken for me - it just spits out "---\n" no matter what prompt I feed it. I got a pelican out of it via the AI Studio API hosted model…

Do you have a single gallery page where we can see all the pelicans together. I'm thinking something similar to https://clocks.brianmoore.com/ but static.

Uh, the GPT-5 clock is... interesting, to say the least.