Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google releases Gemma 4 open models

deepmind.google

Post reply on HN
Print view View original

481–490 of 507 posts

Re: Google releases Gemma 4 open models

#481

Post by nl »

neonstatic wrote: post #478
nl wrote: post #375

Gemma-4-E4B-it scored 15/25 on my https://sql-benchmark.nicklothian.com/#all-data (agentic SQL generation). The naming is a bit odd - E4B is "4.5B effective, 8B with embeddings", so despite the name it is probably best compared with the 8B/9B class models and is competitive with them. Qwen3.5-9B also scores 15/25 in thinking mode for example. The best 9B model I've found is Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distil…

Very happy to see updates to your benchmark. Looking forward to inclusion of larger Gemma 4 models!

The medium one on OpenRouter didn't support tools when I tried it. I will update when there is one.

Re: Google releases Gemma 4 open models

#482

Post by curioussquirrel »

For anyone interested in multilingual performance, which is not usually well benchmarked or reported: Gemma 4 does really well, especially the dense 31B version. In fact, it outperforms many models with an order of magnitude higher number of parameters.

It is not quite capable of performing work on really long tail languages, but their claim of 35 languages supported (and a hint of some knowledge of up to 140) was substantiated by our tests.

If you're doing work outside of English and/or need to run a translation model in your terms, Gemma 4 is a very good candidate.

Re: Google releases Gemma 4 open models

#483

Post by curioussquirrel »

aggregator-ios wrote: post #383

I tested the E2B and E4B models and they get close but inaccurate (non working) results when generating jq queries from natural language. This is of importance to me as I work on https://jsonquery.app and would prefer to use a model that works well with browser inference. gemma-4-26b-a4b-it and gemma-4-31b-it produced accurate results in a few of my tests. But those are 50-60GB in size. Chrome has a developer preview…

Same, I quickly tested it for code gen and it produced mostly good code for simple problems, but it sometimes hallucinated words in non-English scripts inside the code.

Re: Google releases Gemma 4 open models

#484

Post by curioussquirrel »

Retro_Dev wrote: post #359

I'm very pleased with the performance of the largest gemma4 model (which I tested through ollama). My singular data point on whether an LLM remembers things well is whether it can translate toki pona to (and from) English. I find it easy to evaluate because I know the language. This local LLM marks the first version that 1) doesn't hallucinate words - at least, for the largest model - and 2) uses common word-phrases…

We're doing multilingual testing and I can confirm what you've observed: Gemma 4 is surprisingly good at multilingual tasks, especially given its size. This is mostly true for the dense 31B model.

Re: Google releases Gemma 4 open models

#485

Post by curioussquirrel »

vunderba wrote: post #32
originalvichy wrote: post #14

The wait is finally over. One or two iterations, and I’ll be happy to say that language models are more than fulfilling my most common needs when self-hosting. Thanks to the Gemma team!

Strongly agree. Gemma3:27b and Qwen3-vl:30b-a3b are among my favorite local LLMs and handle the vast majority of translation, classification, and categorization work that I throw at them.

Give Gemma 31B a shot for translation, it does a very good job at that given its size.

Re: Google releases Gemma 4 open models

#486

Post by chromatin »

nl wrote: post #375

Gemma-4-E4B-it scored 15/25 on my https://sql-benchmark.nicklothian.com/#all-data (agentic SQL generation). The naming is a bit odd - E4B is "4.5B effective, 8B with embeddings", so despite the name it is probably best compared with the 8B/9B class models and is competitive with them. Qwen3.5-9B also scores 15/25 in thinking mode for example. The best 9B model I've found is Qwen3.5-9B-Claude-4.6-Opus-Reasoning-Distil…

I love that you are doing this test. However, as it purports to be a test of "English-to-SQL", your hardest question (Q9) seems ungrammatical:

> Show order lines, revenue, units sold, revenue per unit (total revenue ÷ total units sold), average list price per product in the subcategory, gross profit, and margin percentage for each product subcategory.

In particular, the clause "in the subcategory, gross profit, and margin percentage for each product subcategory" is ambiguous, and I wonder if more models would pass if the English were reformulated to be correct.

(it's also notable that Claude Opus 4.6 and Sonnet 4.6 both "missed" this one)

Re: Google releases Gemma 4 open models

#487

Post by wok4899 »

evilelectron wrote: post #57
danielhanchen wrote: post #2

Thinking / reasoning + multimodal + tool calling. We made some quants at https://huggingface.co/collections/unsloth/gemma-4 for folks to run them - they work really well! Guide for those interested: https://unsloth.ai/docs/models/gemma-4 Also note to use temperature = 1.0, top_p = 0.95, top_k = 64 and the EOS is " ". " thought\n" is also used for the thinking trace!

Daniel, your work is changing the world. More power to you. I setup a pipeline for inference with OCR, full text search, embedding and summarization of land records dating back 1800s. All powered by the GGUF's you generate and llama.cpp. People are so excited that they can now search the records in multiple languages that a 1 minute wait to process the document seems nothing. Thank you!

This is a very interesting project. If it's publicly available, would you mind sharing it? I would love to understand how it works.

Ps: found your other comments, thanks.

Re: Google releases Gemma 4 open models

#488

Post by om252345 »

Gemma 4 can unlock local agentic coding if coupled with right tools. I feel we need some graph based code kb, external memory and RAG will make it more powerful for local coding. I would say use Claude, gemini for big reactors but for small edits, using gemma 4 should be absolutely fine.

Re: Google releases Gemma 4 open models

#489

Post by XCSme »

5555watch wrote: post #452
XCSme wrote: post #300

Earlier quoted context omitted.

Good work, it's quite close to Gemini 3 Pro in my tests, but 10x cheaper: https://aibenchy.com/compare/google-gemma-4-31b-it-medium/go...

Why no (high) variants in the comparison models?

Good question! I might add them, but there were multiple reasons:

1. Most variants on HIGH/XHIGH provide only marginal improvements in accuracy, but at drastically increased latency and cost. One special example is Gemini 3.1 Flash Lite, which on High used 1.5M reasoning tokens, and it's cost was 5x the one of running 5.3-Codex: https://aibenchy.com/compare/google-gemini-3-1-flash-lite-pr...

2. On medium it seems like most models use a similar amount of reasoning tokens, this should be a more fair comparison.

3. Most models in the wild are used on medium (chat apps, default coding apps, tools, etc.).

4. Running on models on HIGH/XHIGH can lead to huge costs for me maintaining the test suite. I might add more models on high, if I can do it in a sustainable way.

5. Running models on HIGH would make running tests suites take much longer, so the results won't be published as fast.

6. Some models even show degradation when used on HIGH, as they tend to overthink/doubt themselves more. This seems to be a trend especially for new models, which wore trained to actually say "wait, but" quite a lot...

Overall, I am happy with how the current leaderboard/comparisons work. I might test some models on high, but for me, a better indication of true intelligence of a model/AGI is how well it does with "none"/no reasoning, than how well it does with high.

Post reply on HN