Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

301–310 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#301

Post by naasking »

WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

> It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt.

Yes, Gemma 4 is very promising for its strong performance and token efficiency, but it's unfortunate that it's sliding window attention has a fatal flaw that makes me seriously hesitate to rely on it. See the series of videos on this channel:

https://youtu.be/ONQcX9s6_co?si=Yt55_N4DcNLstnGS

On top of Qwen3.5/3.6's superior recall, it's attention mechanism dramatically reduces KV cache requirements, so you can fit longer sessions in the same VRAM (or more concurrent sessions if you have agents running), which is critical for local hosting.

At this point Qwen3.6 with thinking mode disabled seems like the best balance.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#302

Post by mark_l_watson »

kissickas wrote: post #113
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

I only see plans for $8, $20, and $250/month... which one are you using exactly? https://gemini.google/subscriptions/

I was in their Pro plan for about a year, now on Ultra, and I am planning on downgrading to the cheap $8/month plan and just using OpenCode with a good inference provider because of one thing: I like seeing my token and cost $ usage data in real time. I know this sounds a little crazy, but I like visibility into what resources I am using.

I think subscription plans are a little bit evil.

Th said, Ultra with the initial half price deal is awesome: all the Opus tokens I need in AntiGravity.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#303

Post by mark_l_watson »

gertlabs wrote: post #246
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

This is true, we have the numbers to back it up on https://gertlabs.com/rankings?mode=oneshot_coding (check out the efficiency chart too) GPT 5.5/5.4 are the smartest models, but at great token / code bloat cost. Qwen 3.6 Max strikes a good balance. But Gemma 4 26B writes some really efficient code, with great results considering the model size. Things do start falling apart under higher contexts.

I have been experimenting with using Claude Code with both the qwen3.6 31B MOE and 28B dense models. Yesterday the 31B model once got confused on refactoring some Prolog code and took a very long time to get it right. Functionality for coding or refactoring Python or TypeScript is usually good. but runs slowly on my 32B MacMini.

Ollama has initial support for bf16 MTP Gemma 4 https://ollama.com/library/gemma4:31b-coding-mtp-bf16 but I have to wait for a smaller model.

I understand why people get excited by having the strongest AI to play/work with but economic factors of inference really count also.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#304

Post by [deleted] »

gunalx wrote: post #284
cma wrote: post #263

Earlier quoted context omitted.

I think it also gets use in the /fast modes the providers sell at higher cost.

They probably use it on all models. Fast is probably just a resource pool with less congestion and therefore faster throughput per user but less efficent.

[deleted]

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#305

Post by mark_l_watson »

MengerSponge wrote: post #70
Urahandystar wrote: post #60

Earlier quoted context omitted.

True, but you have to add up the cumulative token output if your being fair. That alignment issue requires another set of input and output tokens to correct.

Does it? Or is this a centaur situation where a competent human can fix it in about two minutes?

Yes! Sometimes when models get something wrong with less widely used programming languages, I like to just cancel the current inference, fix something myself, then tell the harness/model that I fixed the current problem, and to move on.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#306

Post by cma »

gunalx wrote: post #284
cma wrote: post #263

Earlier quoted context omitted.

I think it also gets use in the /fast modes the providers sell at higher cost.

They probably use it on all models. Fast is probably just a resource pool with less congestion and therefore faster throughput per user but less efficent.

If it speeds prefill too I guess so.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#308

Post by monster_truck »

lossolo wrote: post #292
mike_hearn wrote: post #275

Earlier quoted context omitted.

Maybe at very high level of abstraction, but there's no branching involved.

Well, there are multiple token proposals processed in parallel, from which only one is picked, seems like branching to me. The only difference is that in case of CPU there is always only one possible branch that is correct.

Well, not exactly, but that was the dream we were sold (here be dragons)

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#309

Post by mark_l_watson »

aleksiy123 wrote: post #66

I’m starting to think that googles strategy is a bit different then the other frontier providers. Focusing more on performance to compute efficiency over pure performance. And maybe that’s why Gemini is (seemingly) lagging behind? Other providers hitting capacity and hitting the limits subsidising their inference. Google strategy seems to be about scaling and distributing these models to their existing billions of us…

I like Google’s business model more than the other frontier model providers: sustainable. One thing I don’t like with Gemini Ultra is no visibility into token use or what the cost would be. I have been planning on letting my Ultra subscription expire and go with OpenCode with a fast inference provider to get this visibility, but this discussion thread gave me the idea of also trying the paid APIs with AntiGravity instead of a subscription. When I sit down to do a specific task I want accurate token usage and $$ data as I work.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#310

Post by intothemild »

apexalpha wrote: post #289
sigmoid10 wrote: post #123

Earlier quoted context omitted.

Gemma certainly was trained for tool calling, but the implementation in llama.cpp has been troubled because Gemma uses a different chat template format. The processor from the transformers library works fine though.

Oh I must've missed this. The AI space moves so fast! I'll check it out again.

Don't forget to update the gguf you have too. The templates in them were updated recently too
Post reply on HN