Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

121–130 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#121

Post by zobzu »

SwellJoe wrote: post #108
ActorNightly wrote: post #52

Earlier quoted context omitted.

Qwen is still better that Gemma though. Also you can tune it more for different tasks, which means that you can prioritize thinking and accuracy versus inference speed.

Qwen is better at some things (code, in particular), but Gemma has better prose and better vision. At least, it feels that way to me.

gemma is also just way faster. i dont wanna wait 10min to get a 5-10% better answer (and sometimes, actually worse answer).

best is to use your own model router atm, depending on the task

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#122

Post by redman25 »

ActorNightly wrote: post #52
msp26 wrote: post #43

Google is singlehandedly carrying western open source models. Gemma 4 31B is fantastic. However, it is a little painful to try to fit the best possible version into 24GB vram with vision + this drafter soon. My build doesn't support any more GPUs and I believe I would want another 4090 (overpriced) for best performance or otherwise just replace it altogether.

Qwen is still better that Gemma though. Also you can tune it more for different tasks, which means that you can prioritize thinking and accuracy versus inference speed.

It’s a heck of a lot faster too.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#123

Post by sigmoid10 »

apexalpha wrote: post #114
nzeid wrote: post #91

Earlier quoted context omitted.

A few days ago I switched again from Qwen3.6 to Gemma 4 - for personal use I've experienced better average performance with the 26B version of the latter than the 27B of the former. For someone who's been running local models for a long while, these are very very exciting times.

I’ve been swapping between these too as well. However I find qwen unbeatable for toolcallling. I think gemma wasnt trained on that at all.

Gemma certainly was trained for tool calling, but the implementation in llama.cpp has been troubled because Gemma uses a different chat template format. The processor from the transformers library works fine though.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#124

Post by logickkk1 »

deskamess wrote: post #30

Did DeepSeek come up with MTP? It was listed prominently in their recent paper as being carried forward from the previous release.

i think this is mixing two separate ideas. MTP is the training-side piece. speculative decoding is the inference trick. DeepSeek V3 used MTP as an auxiliary loss. the 2022 Google paper is speculative decoding. now Google is combining them. https://arxiv.org/abs/2404.19737

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#125

Post by lucb1e »

rjh29 wrote: post #59
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for this assist function; in the best region (a few counties currently running on 100% hydropower) it makes no difference of course because the energy is running down the hill whether you use it or not. Maybe it could otherwise have been exported or stored but there's only so much interconnect and storage

Edit: and this 15$ subscription (again assuming 225×8h use per year divided by 12 months) uses the equivalent of about 150€/month worth of electricity at the rate I'd pay at home. That sounds close to the cost price (ignoring capex on the servers and model training) Google would be able to negotiate with electricity providers. Would be interested in how this works out for them if someone knows

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#126

Post by adrian_b »

julianlam wrote: post #83

Does this mean there will be new Gemma 4 models released with MTP, or are they already available in existing models + quants?

For each of the 4 gemma-4-*-it models there has been published an associated small model gemma-4-*-it-assistant, to be used for MTP.

If a GGUF file is generated for MTP, it must include both the big model and the small model. There was a reference in another comment to a PR for llama.cpp, which also included updates for the Python program used for conversion from the safetensors files, which presumably can handle the combining of the two paired Gemma 4 models.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#127

Post by jfim »

basch wrote: post #54
zdw wrote: post #9

MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.

I have a dumb performance question. Why when asking a model to change text in a minor way; are we not asking it to generate the operational transformations necessary to modify the text, and then just executing the ot on the existing text vs reproducing every token? Maybe tools are doing that more than I realize?

I've seen Claude use sed to edit files on other hosts instead of copying the file back and forth to edit it. Not quite full blown OT but it's going in that direction.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#128

Post by petu »

furyofantares wrote: post #102
Klaus23 wrote: post #94

Earlier quoted context omitted.

It really is. This is because LLMs with a single output/user are strongly bandwidth limited. Although the hardware can generate multiple tokens simultaneously, it is slowed down if the tokens depend on each other, as is the case with regular text generation. The draft model essentially predicts the next token quickly, enabling you to start generating the subsequent token in parallel. If the guess is right, the second…

> If the guess is right This is the crux. What makes the guess "right"? I think the acceptance criteria is not that the token is exactly the token the big model would have produced. It's accepted of the big model verifies that the probability of that token was high enough. How close it is to the same output (or same distribution of outputs) you'd get from running the big model would be dependent on temperature, top-k…

> What makes the guess "right"?

Matching token that would've been picked without speculative decoding. That seems to be more or less agreed upon.

e.g. vLLM docs list tests they run to ensure that output doesn't change if spec. decoding is used: https://github.com/vllm-project/vllm/blob/main/docs/features...

But introducing some threshold to accept other high probability tokens is interesting idea.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#129

Post by nzeid »

apexalpha wrote: post #114
nzeid wrote: post #91

Earlier quoted context omitted.

A few days ago I switched again from Qwen3.6 to Gemma 4 - for personal use I've experienced better average performance with the 26B version of the latter than the 27B of the former. For someone who's been running local models for a long while, these are very very exciting times.

I’ve been swapping between these too as well. However I find qwen unbeatable for toolcallling. I think gemma wasnt trained on that at all.

I'm using llama.cpp with Gemma and tool calling is mission critical. It's perfectly fine on my end.

There are definitely differences in the eagerness to tool-call that you'll need to manage. And for all local models I've ever used, I've had to micromanage the tools provided by servers to eliminate any possibility that they reach for something wonky or confusing.

Post reply on HN