Google releases Gemma 4 open models • Hacker News Board

Re: Google releases Gemma 4 open models

#361

Post by trashcan2137 » Fri, Apr 03, 2026, 2:05 AM UTC

danielhanchen wrote: post #2

Thinking / reasoning + multimodal + tool calling. We made some quants at https://huggingface.co/collections/unsloth/gemma-4 for folks to run them - they work really well! Guide for those interested: https://unsloth.ai/docs/models/gemma-4 Also note to use temperature = 1.0, top_p = 0.95, top_k = 64 and the EOS is " ". " thought\n" is also used for the thinking trace!

  and the EOS is "". "thought\n" is also used for the thinking trace!

Can someone explain this to me? Why is this faux-XML important here?

Re: Google releases Gemma 4 open models

#362

Post by why_only_15 » Fri, Apr 03, 2026, 2:07 AM UTC

c0wb0yc0d3r wrote: post #354

yags wrote: post #348

Earlier quoted context omitted.

We (LM Studio) found the bug with the 31B model and a fix will be going out hopefully tonight

I am not deep in this world. What does it mean when you (LM Studio) fixed a bug in a model Google released?

I am in this world, but am not familiar with this specifically.

My guess is that they found a bug with their implementation of the model using the weights Google released. These bugs are often difficult to track down because the only indication is that the model is worse with your implementation than with someone else's.

Re: Google releases Gemma 4 open models

#363

Post by Havoc » Fri, Apr 03, 2026, 2:08 AM UTC

simonw wrote: post #77

I ran these in LM Studio and got unrecognizable pelicans out of the 2B and 4B models and an outstanding pelican out of the 26b-a4b model - I think the best I've seen from a model that runs on my laptop. https://simonwillison.net/2026/Apr/2/gemma-4/ The gemma-4-31b model is completely broken for me - it just spits out "---\n" no matter what prompt I feed it. I got a pelican out of it via the AI Studio API hosted model…

Same experience on the 31B - something’s wrong. The MoE works as expected though.

Re: Google releases Gemma 4 open models

#364

Post by burgerquizz » Fri, Apr 03, 2026, 2:10 AM UTC

I want to embed a lightweight local model to be used for my webapp to use it without thinking about token price. is there an acceptable way to do it today?

Re: Google releases Gemma 4 open models

#365

Post by mybigbro » Fri, Apr 03, 2026, 2:13 AM UTC

Went through the official blog and the developers post, no mention of TurboQuant anywhere. Google's own research team tested it on Gemma models for KV-cache compression to 3 bits, so it's surprising it's not mentioned in this release. Anyone know if it's baked in already or if we'd need to apply it ourselves? Would love to run the 26B MoE locally as a daily driver.

Re: Google releases Gemma 4 open models

#366

Post by vessenes » Fri, Apr 03, 2026, 2:14 AM UTC

zozbot234 wrote: post #204

vessenes wrote: post #195

Earlier quoted context omitted.

I’ve been doing some low-key testing on smaller models, and it looks to me like it’s possible to train an MOE model with characteristics that are helpful for streaming… For instance, you could add a loss function to penalize expert swapping both in a single forward, pass and across multiple forward passes. So I believe there is a place for thinking about this on the model training side.

Penalizing expert swaps doesn't seem like it would help much, because experts vary by layer and are picked layer-wise. There's no guarantee that expert X in layer Y that was used for the previous token will still be available for this token's load from layer Y. The optimum would vary depending on how much memory you have at any given moment, and such. It's not obviously worth optimizing for.

Right. You need to predict a set of experts through the entire forward pass. Think of a vertical strip.

Re: Google releases Gemma 4 open models

#367

Post by genpfault » Fri, Apr 03, 2026, 2:14 AM UTC

danielhanchen wrote: post #2

Thinking / reasoning + multimodal + tool calling. We made some quants at https://huggingface.co/collections/unsloth/gemma-4 for folks to run them - they work really well! Guide for those interested: https://unsloth.ai/docs/models/gemma-4 Also note to use temperature = 1.0, top_p = 0.95, top_k = 64 and the EOS is " ". " thought\n" is also used for the thinking trace!

llama.cpp (b8642) auto-fits ~200k context on this 24GB RX 7900 XTX & it shows a solid 100+ tok/s ("S_TG t/s") on the first 32k of it, nice!

    ./llama-batched-bench -hf unsloth/gemma-4-26B-A4B-it-GGUF:UD-Q4_K_XL \
    -npp 1000,2000,4000,8000,16000,32000,64000,96000,128000 -ntg 128 -npl 1 -c 0
    |    PP |     TG |    B |   N_KV |   T_PP s | S_PP t/s |   T_TG s | S_TG t/s |      T s |    S t/s |
    |-------|--------|------|--------|----------|----------|----------|----------|----------|----------|
    |  1000 |    128 |    1 |   1128 |    0.416 |  2404.87 |    1.064 |   120.29 |    1.480 |   762.20 |
    |  2000 |    128 |    1 |   2128 |    0.755 |  2649.86 |    1.075 |   119.04 |    1.830 |  1162.83 |
    |  4000 |    128 |    1 |   4128 |    1.501 |  2665.72 |    1.093 |   117.08 |    2.594 |  1591.49 |
    |  8000 |    128 |    1 |   8128 |    3.142 |  2545.85 |    1.114 |   114.87 |    4.257 |  1909.47 |
    | 16000 |    128 |    1 |  16128 |    6.908 |  2316.00 |    1.189 |   107.65 |    8.097 |  1991.73 |
    | 32000 |    128 |    1 |  32128 |   16.382 |  1953.31 |    1.278 |   100.12 |   17.661 |  1819.16 |
    | 64000 |    128 |    1 |  64128 |   43.427 |  1473.74 |    1.453 |    88.12 |   44.879 |  1428.89 |
    | 96000 |    128 |    1 |  96128 |   82.227 |  1167.50 |    1.623 |    78.86 |   83.850 |  1146.42 |
    |128000 |    128 |    1 | 128128 |  133.237 |   960.69 |    1.797 |    71.25 |  135.034 |   948.86 |

Re: Google releases Gemma 4 open models

#368

Post by car » Fri, Apr 03, 2026, 2:15 AM UTC

Wowfunhappy wrote: post #262

danielhanchen wrote: post #2

Thinking / reasoning + multimodal + tool calling. We made some quants at https://huggingface.co/collections/unsloth/gemma-4 for folks to run them - they work really well! Guide for those interested: https://unsloth.ai/docs/models/gemma-4 Also note to use temperature = 1.0, top_p = 0.95, top_k = 64 and the EOS is " ". " thought\n" is also used for the thinking trace!

Hi! Do you ever make quants of the base models? I'm interested in experimenting with them in non-chat contexts.

Yes, they are listed on huggingface. The instruction trained models have an 'it' in their name.

https://huggingface.co/collections/unsloth/gemma-4

Edit: Sorry, I'm not sure if this is a quant, but it says 'finetuned' from the Google Gemma 4 parent snapshot. It's the same size as the UD 8-bit quant though.

Re: Google releases Gemma 4 open models

#369

Post by i386 » Fri, Apr 03, 2026, 2:18 AM UTC

You can try this new model live using mesh-llm right now: https://www.anarchai.org/dashboard

Re: Google releases Gemma 4 open models

#370

Post by vparseval » Fri, Apr 03, 2026, 2:28 AM UTC

NekkoDroid wrote: post #100

WarmWash wrote: post #86

Earlier quoted context omitted.

Mainline consumer cards are 16GB, so everyone wants models they can run on their $400 GPU.

Yea, I've been waiting a while for a model that is ~12-13GB so there is still a bit of extra headroom for all the different things running on the system that for some reason eat VRAM.

I found that you can run models locally pretty well that exceed your VRAM by a bit. At least ollama will hand excess off to your system RAM. Maybe performance suffers but I've never actually seen it crap out and I can wait a few minutes for a response.