Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

231–240 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#231

Post by prodigycorp »

WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

I think you can see this one of two ways: you could also consider it a miracle that the qwen models are able to perform so well when being trained on inefficient wrapper code data.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#232

Post by prodigycorp »

rjh29 wrote: post #59
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

This used to be the case, but the changes last month have rendered the Gemini Pro plan completely unusable.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#233

Post by imrozim »

ydj wrote: post #229
imrozim wrote: post #228

3x faster inference means cheaper api costs tooo. For solo dev building ai this matters a lot

Not necessarily. Servers serving the model likely has enough traffic that they are batching decodes already. MTP reduces latency and increase efficiency only when the server can’t batch enough concurrent streams to be compute bound rather than memory bound.

Fair didn't think about batching makes more sense for self hosted models then.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#234

Post by m3kw9 »

WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

it won't really do much if you try to code with it. i plugged it into xcode and it failed to change a variable.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#235

Post by 0xbadcafebee »

Zarathruster wrote: post #68
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

Where are you using it? Is Gemini CLI at a usable state? It was a frustrating, miserable experience last time I gave it a shot. Antigravity seems significantly better in comparison, but with lower usage limits. If I run out, I usually don't bother switching to Gemini CLI.

> Is Gemini CLI at a usable state?

Technically usable but with bad/broken code. I found 3 different bugs with 1 feature, found a duplicate feature (their vibe coding missed the fact that the feature was already implemented), and the docs were wrong. Other features were ridiculously badly implemented. Reported them all, submitted multiple changes. None were accepted. Their repo was a hellscape of AI-generated issues and AI-generated PRs; I think mine was the only one written by a human. This was a month and a half ago.

Google is one of the most valuable corporations in the world, yet even they shipped a turd of an app to real customers and can't even take a bug fix. I think AI coding might be cooked.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#236

Post by julianlam »

adrian_b wrote: post #106
el_isma wrote: post #96

How is this different from the speculative decoding that we had before? You could pair a big and small model like qwen 32b with qwen 4b and had that same dynamic of the small model generating tokens and the big one "certifiying" them. The blog says something about re-using the big model's data?

Multi token prediction is the same thing as speculative decoding. This is mentioned in the Google pages describing their MTP implementation. Google has now provided small models for each of the previous Gemma 4 models, e.g. "gemma-4-26B-A4B-it-assistant" for "gemma-4-26B-A4B-it". The difference vs. Qwen is that here each small model is not some general-purpose smaller model, but a model that has been optimized specif…

So then these models could be used by llama.cpp today with the -md switch?

Interesting, must try tomorrow.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#237

Post by libraryofbabel »

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the checks in parallel. And if you think about it, a lot of next tokens are pretty obvious in certain situations (e.g. it doesn't take a frontier model to guess the likely next token in "United States of...", and a lot of code is boilerplate and easy to predict from previous code sections).

I always encourage folks who are interested in LLM internals to read up on speculative decoding (both the basic version and the more advanced MTP), and if you have time, try and implement your own version of it (writing the core without a coding agent, to begin with!)

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#239

Post by qingcharles »

xnx wrote: post #223
qingcharles wrote: post #215

Earlier quoted context omitted.

Also bonus: using it in Antigravity you can burn through all the Opus credit Google give you first to do all the planning and then switch it to Gemini 3.1 Pro to do the grunt work.

Have you compared Opus and Gemini to see if Gemini is any worse at planning than Opus?

Yes, Gemini 3.1 Pro (High) is still inferior to Opus 4.6 (Thinking) that Google are offering, for planning. It just doesn't think things through as thoroughly as Opus. I'll use it when I've burned up all my Opus tokens and I still have planning I want to do, but I'll read the plan very carefully, whereas with Opus I'll only give it a cursory scan through.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#240

Post by xnx »

qingcharles wrote: post #239
xnx wrote: post #223

Earlier quoted context omitted.

Have you compared Opus and Gemini to see if Gemini is any worse at planning than Opus?

Yes, Gemini 3.1 Pro (High) is still inferior to Opus 4.6 (Thinking) that Google are offering, for planning. It just doesn't think things through as thoroughly as Opus. I'll use it when I've burned up all my Opus tokens and I still have planning I want to do, but I'll read the plan very carefully, whereas with Opus I'll only give it a cursory scan through.

Good data point. I would venture 90+% of Claude users have dismissed Gemini without every trying it.
Post reply on HN