Gemma 4 12B: A unified, encoder-free multimodal model • Hacker News Board

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#391

Post by pja » Thu, Jun 04, 2026, 8:30 PM UTC

superkuh wrote: post #213

senko wrote: post #120

I ran the Q4 quant (used with llama.cpp) though my "minesweeper" vibe-coding benchmark: https://senko.net/vibecode-bench/2026/minesweeper-gamma-4-12... The result is decent, but it had a few bizzare/trivial syntax errors I had to fix manually: it would do an extra closing bracket or paren a few times, and wanted to separate function definitions with comma. Not sure what that was about, but otherwise the output run ju…

>consumer-grade card with 12G of VRAM and got 5t/s That speed for token output indicates to me that it somehow is using hybrid mode and involving cpu+system ram somehow. That ~5tk/s is about the ram bandwidth of DDR4 RAM versus that size model at 4bit. Any consumer GPU with 12 GB like a nvidia rtx 2080 or rtx 3060 should be doing 20+ tk/s with llama.cpp and CUDA backend.

The 8 bit quant runs at 36tps using Vulkan on my AMD rx9070.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#392

Post by pja » Thu, Jun 04, 2026, 8:34 PM UTC

senko wrote: post #120

I ran the Q4 quant (used with llama.cpp) though my "minesweeper" vibe-coding benchmark: https://senko.net/vibecode-bench/2026/minesweeper-gamma-4-12... The result is decent, but it had a few bizzare/trivial syntax errors I had to fix manually: it would do an extra closing bracket or paren a few times, and wanted to separate function definitions with comma. Not sure what that was about, but otherwise the output run ju…

The Unsloth 8bit quant seems to manage this task without any syntax errors.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#393

Post by ilovecake1984 » Thu, Jun 04, 2026, 8:41 PM UTC

dofm wrote: post #127

minimaxir wrote: post #2

The big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is ro…

I would contend that the actual big story is the gallery app: https://developers.google.com/edge/gallery Anyone with a 16GB Mac — that is quite a lot of journalists, surely — can download that, install a model into it, and play. Surely journalists have to start asking questions at least about OpenAI's consumer revenue projections now. I am a major, major AI cynic, but I decided to be an informed cynic so I've been pl…

Ai is not investable. It is useful.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#394

Post by yassa9 » Thu, Jun 04, 2026, 8:51 PM UTC

sgt101 wrote: post #388

0xbadcafebee wrote: post #206

Earlier quoted context omitted.

It was almost certainly not trained for coding, as it's got both audio and vision input, is only 12B, and nowhere in the announcement is coding mentioned. It will likely not have good performance on coding in general, compared to other small models like Qwen 3.6 35B A3B, Gemma 4 26B A4B, Nvidia Nemotron 3 Nano 30B-A3B, gpt-oss-20b. For 16GB laptops, Qwen 3.5 9B is the undisputed champ. Gemma 4 31B is the top dog at s…

31B won't run in 48GB for me - it needs 54.

what quantization did u try ? u can use Q4 quantization, im pretty sure that 48GB would be enough

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#395

Post by yassa9 » Thu, Jun 04, 2026, 8:58 PM UTC

goobatrooba wrote: post #165

minimaxir wrote: post #2

The big story here is the encoder-free part, which I still don't fully understand. > Vision: We replaced Gemma 4’s vision encoder with a lightweight embedding module consisting of a single matrix multiplication, positional embedding and normalizations. That's technically encoding, just without using a dedicated model for it like SigLIP? The Developer's Guide elaborates, it's still a 35M layer which I am curious is ro…

Either Google changed the text or you editorialised it a tiny bit - just for all others that got excited, they mean 16GB V RAM. So a premium graphics card requiring a >2500€ device is the minimum to run this. Still progress, but not quite democratic yet. Weird though that Google might be cannibalising it's own AI subscription service?

RTX 5060 ti 16gb is about ~550-600 usd, even u cant get AMD RX 9060 XT 16gb with only 400 usd, so why 2500€

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#396

Post by burnte » Thu, Jun 04, 2026, 9:08 PM UTC

asim wrote: post #137

We are now entering the closed loop game. Google doesn't need anyone else to accelerate their models. This is their bread and butter. I'm both shocked but also not surprised that they continue to develop such efficiencies. Honestly it's like silicon and CPU architecture advancement. We kept shrinking it and shrinking it and it kept getting more and more powerful and here we are with AI and it's only going to be 100x…

> Beyond our lifetimes people will live to 125 quite steadily and with great mobility

Possibly, but it'll take a lot of bioengineering. It looks like our metabolic processes really tap out at 120, so we'll need some way to maintain better biological condition body-wide at earlier ages so we're "younger" for longer. Definitely possible though, just hard.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#397

Post by fluffyspork » Thu, Jun 04, 2026, 9:18 PM UTC

According to Gemini to run full 256k context window with unified RAM

4-bit Quantized (Q4_K_M GGUF): You need at least 25 GB of total RAM. This is the most practical configuration for consumer hardware.

8-bit Quantized (Q8_0 / SFP8): You need at least 32 GB to 36 GB of RAM

Uncompressed 16-bit (BF16): You will need upwards of 45 GB to 50 GB of RAM to account for both the 26.7 GB base model and the massive KV cache.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#398

Post by kanemcgrath » Thu, Jun 04, 2026, 9:33 PM UTC

ricardobayes wrote: post #363

dirkg wrote: post #326

Earlier quoted context omitted.

> For 16GB laptops, Qwen 3.5 9B is the undisputed champ. you can run qwen 3.6 35BA3B on a 12-16GB vram gpu and ot works pretty well. https://www.youtube.com/watch?v=8F_5pdcD3HY&t=1s even the 27B in some quants can fit. https://www.reddit.com/r/LocalLLaMA/comments/1tkmgwj/qwen27b... qwen IMO is far better for coding, esp agentic coding when combined with something like Pi, it comes probably close enough to Sonnet for…

You can run it, however those low quantized models (iQ2, iQ4, Q2) will very likely underperform the 9B versions at Q6/Q8.

Something about qwen models hold up really well even at low quants. for most other models anything under q5 is cooked, but on 35B-A3B I can get a lot of things done even at q3_xl. It is definitely better than full precision 9B

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#399

Post by rft » Thu, Jun 04, 2026, 9:37 PM UTC

fluffyspork wrote: post #397

According to Gemini to run full 256k context window with unified RAM 4-bit Quantized (Q4_K_M GGUF): You need at least 25 GB of total RAM. This is the most practical configuration for consumer hardware. 8-bit Quantized (Q8_0 / SFP8): You need at least 32 GB to 36 GB of RAM Uncompressed 16-bit (BF16): You will need upwards of 45 GB to 50 GB of RAM to account for both the 26.7 GB base model and the massive KV cache.

Those numbers are wrong for most use cases. Likely the LLM did not take SWA (Sliding Window Attention) of G4 into account. Without SWA those numbers could be correct, I can't load a q8 without SWA on a 24GB card.

I tested this with the at the time newest llama.cpp master on a Linux system with 2 3090 24GB, only one was used for testing. q8 without any KV quant, 256k context, mmproj loaded takes less than 20GB VRAM. This runs at about 1.5k to 2k tok/s pp and 40-50 tok/s gen (slightly lowered power limits & undervolted). q8 with 64k non-quant context and mmproj takes just under 16GB VRAM. Drop down to the q6k model, no mmproj, 64k non-quant context and it fits in 12GB VRAM. All the way down to q4km and some batch size tweaking and it barely fits into 8GB VRAM.

64k context is the minimum for Hermes agent, so a vision capable "agentic" model fits into a 16GB card. This is very impressive. I am currently testing how smart the model is and it does decently so far, had one looping issue it recovered after a lot of tokens, did some basic tool calling.

Re: Gemma 4 12B: A unified, encoder-free multimodal model

#400

Post by avadodin » Thu, Jun 04, 2026, 10:38 PM UTC

thot_experiment wrote: post #179

brianwawok wrote: post #34

Earlier quoted context omitted.

Every other Google model I have tried felt very weak compared to qwen models. I dont have a ton of use case for multimodal though, so its very possible this is a fantastic multimodal model.

Hard disagree, Qwen multimodal is way better than google's, but Gemma 31b runs laps around Qwen 27B in complex engineering tasks. Maybe Qwen is better at slopcoding web framework CRUD, but for embedded dev there's no comparison.

E4B is decent at instruction following. It managed to produce a deliverable on par with the lowest tier of paid models. Even higher tiers often just ignore all rules when they feel like it.

I wish it was an 8BA1B MoE model with the newer acceleration 1B or maybe even a tailor-made sub-1B slapped on top. That would make it an awesome local model for the average laptop.