Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

281–290 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#281

Post by mike_hearn »

2ndorderthought wrote: post #90
tln wrote: post #86

Earlier quoted context omitted.

Taalas. A sibling comment of yours posted the chat demo URL - https://chatjimmy.ai/

Woah. How is this working? It's stupid fast.

The weights are mapped directly to transistors. It's not a generic processor, it's literally a dedicated Llama 8B chip that can't be used for anything else. When you specialize in hardware you get faster - Taalas is pushing that to the limit.

They seem to be doing well. I checked recently and their API is closed to signups due to overwhelming demand.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#282

Post by janalsncm »

zmmmmm wrote: post #267
libraryofbabel wrote: post #237

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the…

> it's fast to check that they are actually correct with the main model because you can run the checks in parallel. Can you give an intuition as to why it's faster? I would have thought regardless how many you run in parallel, the successful check has to execute the full model to generate the full sequence so you will have exactly the same time needed? Or is it by process of elimination so it terminates early once it…

The small draft model proposes a sequence of tokens d1 d2 d3.

The big target model calculates

P(d1)

P(d2|d1)

P(d3|d1 d2)

In parallel. If we were just greedy decoding it would be simple. Just stop when the draft model doesn’t predict the most likely token as judged by the target model. At that point, append the correct token from the target model and kick off both models again in parallel.

In practice we aren’t using greedy decoding. We are sampling and we need to match the target model’s distribution. To do this, we accept tokens from the draft model probabilistically, which is possible because we have the logits of both the draft model and the target at that point. The ratio of their softmax probabilities is used for this.

You are right that actually accepting tokens has to happen sequentially but that’s a heck of a lot faster than a forward pass.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#284

Post by gunalx »

cma wrote: post #263
zozbot234 wrote: post #248

Earlier quoted context omitted.

> But I think the key is that in the standard autoregressive case we get memory bandwidth bound, so there are tons of idle compute resources. Right, this is the same way batching works. It's "free" until we exhaust available compute resources, at which point decode throughput becomes compute bound. (This is a good place to be, because scaling out compute is a lot easier than adding fast VRAM.) This is why MTP is most…

I think it also gets use in the /fast modes the providers sell at higher cost.

They probably use it on all models. Fast is probably just a resource pool with less congestion and therefore faster throughput per user but less efficent.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#285

Post by zmmmmm »

janalsncm wrote: post #282
zmmmmm wrote: post #267

Earlier quoted context omitted.

> it's fast to check that they are actually correct with the main model because you can run the checks in parallel. Can you give an intuition as to why it's faster? I would have thought regardless how many you run in parallel, the successful check has to execute the full model to generate the full sequence so you will have exactly the same time needed? Or is it by process of elimination so it terminates early once it…

The small draft model proposes a sequence of tokens d1 d2 d3. The big target model calculates P(d1) P(d2|d1) P(d3|d1 d2) In parallel. If we were just greedy decoding it would be simple. Just stop when the draft model doesn’t predict the most likely token as judged by the target model. At that point, append the correct token from the target model and kick off both models again in parallel. In practice we aren’t using…

nice ... i think i get the idea - it's effectively the same / similar benefit as batching, but you're batching against your own speculated future path. Which would be pointless if you didn't have a high probability path to evaluate against - but the draft gives you that.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#286

Post by avadodin »

maille wrote: post #177
jszymborski wrote: post #29

Earlier quoted context omitted.

The A4B model is blazing fast and the model is super good at general inquiries. Notably worse than Qwen 3.6 for coding tasks but that says more about the Qwen model.

Bad at coding, but would it be good at code review?

Good compared to what? Nothing? Probably better.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#287

Post by rjh29 »

0xbadcafebee wrote: post #235
Zarathruster wrote: post #68

Earlier quoted context omitted.

Where are you using it? Is Gemini CLI at a usable state? It was a frustrating, miserable experience last time I gave it a shot. Antigravity seems significantly better in comparison, but with lower usage limits. If I run out, I usually don't bother switching to Gemini CLI.

> Is Gemini CLI at a usable state? Technically usable but with bad/broken code. I found 3 different bugs with 1 feature, found a duplicate feature (their vibe coding missed the fact that the feature was already implemented), and the docs were wrong. Other features were ridiculously badly implemented. Reported them all, submitted multiple changes. None were accepted. Their repo was a hellscape of AI-generated issues a…

It's a vibe coded mess, really depressing from such a large company. You can tell it's AI-driven because they keep adding new useless features but not improving the UX or bug fixing the existing ones.

One simple example is you can use @ to reference filenames - but the file list is cached and never updates. Ask Gemini to split a file into two files, then type @ and the new files will never appear. Those kind of extremely basic bugs.

But hey, the text has gradient colours...

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#288

Post by dchftcs »

el_isma wrote: post #96

How is this different from the speculative decoding that we had before? You could pair a big and small model like qwen 32b with qwen 4b and had that same dynamic of the small model generating tokens and the big one "certifiying" them. The blog says something about re-using the big model's data?

It's the same speculative decoding. The news is that it came out for a popular local model.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#289

Post by apexalpha »

sigmoid10 wrote: post #123
apexalpha wrote: post #114

Earlier quoted context omitted.

I’ve been swapping between these too as well. However I find qwen unbeatable for toolcallling. I think gemma wasnt trained on that at all.

Gemma certainly was trained for tool calling, but the implementation in llama.cpp has been troubled because Gemma uses a different chat template format. The processor from the transformers library works fine though.

Oh I must've missed this.

The AI space moves so fast! I'll check it out again.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#290

Post by Klaus23 »

furyofantares wrote: post #186
Klaus23 wrote: post #143

Earlier quoted context omitted.

The token is correct if it matches the one generated by the main model. It works like this: The draft model quickly generates draft-token 1. The main model then starts working on two tokens in parallel. It calculates token 1 based on the context, and token 2 based on the context + draft-token 1. Once the two tokens have been generated, you can check whether the draft-token 1 from the draft model matches token 1 from…

Models do not generate tokens. They generate probabilities for each token. Inference parameters select a token using those. You can just select the top token all the time or you can do it probabilistically. How you do that in both the speculative decoding and the main inference changes how likely you get the exact same tokens. And then you can choose to accept only if the token matches exactly, or you can choose to a…

In theory, you could do that and increase the speed at higher temperatures, but it would subtly alter your output based on the draft model preferences. Rather than picking randomly from the main model probabilities, you would have to accept a draft model pick if it is close enough.

As far as I know, this is not used in practice. Currently popular implementations always match the main model output, and the draft model only affects the speed.

Post reply on HN