Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

311–320 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#311

Post by WarmWash »

amunozo wrote: post #277
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Gemini models, even if not so good at coding, are also competitive with GPT-5.5 and Claude Opus 4.7 in a lot of tasks while having considerably less parameters.

Outside of programming, I haven't gotten a good response from Opus (4.6 or 4.7). Optics, finance, and economics questions. All had glaring oversights. 5.5 is the strongest and very thorough. 3.1 comes very close, and while less thorough, it completes the response in Which begs the question, where would 3.1 be if google let it run for 20 minutes on a prompt? Possibly worse, but you have to wonder.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#312

Post by furyofantares »

Klaus23 wrote: post #290
furyofantares wrote: post #186

Earlier quoted context omitted.

Models do not generate tokens. They generate probabilities for each token. Inference parameters select a token using those. You can just select the top token all the time or you can do it probabilistically. How you do that in both the speculative decoding and the main inference changes how likely you get the exact same tokens. And then you can choose to accept only if the token matches exactly, or you can choose to a…

In theory, you could do that and increase the speed at higher temperatures, but it would subtly alter your output based on the draft model preferences. Rather than picking randomly from the main model probabilities, you would have to accept a draft model pick if it is close enough. As far as I know, this is not used in practice. Currently popular implementations always match the main model output, and the draft model…

Here is the line in vLLM's source code that determines if a draft token is accepted:

    accepted = draft_prob > 0 and target_prob / draft_prob >= uniform_prob
It does have a branch that checks only token id equality, which is used if temperature is 0.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#313

Post by asdfasgasdgasdg »

rjh29 wrote: post #255
asdfasgasdgasdg wrote: post #200

Earlier quoted context omitted.

I'm using it in antigravity, and fint it quite good. I have not managed to run out of usage on Flash. You can run Pro out of quota almost instantly, they really don't want you to use it if you're not paying $200 a month. I do not use super broad prompts, though. None of this "build me a webapp" stuff. It's more like, "adjust this part of this class to do Y instead of X."

If you use the Pro model, it can handle fairly broad prompts. Flash is very basic (no thinking)

Sure, but with the $15/mo plan you run out of pro so fast that I prefer not to rely on it. I'll do broader prompts in two years when the cheap models are as smart as the frontier models are today.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#314

Post by ctbellmar »

vparseval wrote: post #210
SwellJoe wrote: post #133

Earlier quoted context omitted.

I'm pretty sure Qwen is faster? The MoE version of Qwen is 3B active, while Gemma 4 is 4B active. Similarly, the dense Qwen is 27B while Gemma is 31B. All else being equal (though I know all else isn't equal), Qwen should be faster in both cases. I haven't actually measured with any precision, but on my AMD hardware (Strix Halo or dual Radeon Pro V620) they seem quite similar in both cases...both MoE models are fast…

qwen-3.6 is really interesting. The dense 27B model is pretty slow for me whereas the sparse 31B is blazingly fast but it also needs to be since it's so chatty. It produces pages and pages of stream of consciousness stuff. 27B does this to a lesser extent but slow enough that I can actually read it whereas 31B just blasts by. I haven't yet compared either to Gemma 4. I tried that out the day after it came out with th…

I had the same experience with 31B. Runs well on 4090 too!

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#315

Post by Klaus23 »

furyofantares wrote: post #312
Klaus23 wrote: post #290

Earlier quoted context omitted.

In theory, you could do that and increase the speed at higher temperatures, but it would subtly alter your output based on the draft model preferences. Rather than picking randomly from the main model probabilities, you would have to accept a draft model pick if it is close enough. As far as I know, this is not used in practice. Currently popular implementations always match the main model output, and the draft model…

Here is the line in vLLM's source code that determines if a draft token is accepted: accepted = draft_prob > 0 and target_prob / draft_prob >= uniform_prob It does have a branch that checks only token id equality, which is used if temperature is 0.

Good analysis. That's surprising. I always heard that the draft model doesn't affect the output in any way. It seems they do it like this to achieve faster generation. It would be interesting to investigate how this affects the output.

Edit: I haven't gone through all the code, but they might do something like this: https://arxiv.org/abs/2211.17192 where a draft model is used and the output distribution is tweaked on rejection, resulting in the exact same distribution as the main model.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#316

Post by 2ndorderthought »

mike_hearn wrote: post #281
2ndorderthought wrote: post #90

Earlier quoted context omitted.

Woah. How is this working? It's stupid fast.

The weights are mapped directly to transistors. It's not a generic processor, it's literally a dedicated Llama 8B chip that can't be used for anything else. When you specialize in hardware you get faster - Taalas is pushing that to the limit. They seem to be doing well. I checked recently and their API is closed to signups due to overwhelming demand.

I want to buy a chip not API access!!!

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#317

Post by sporkland »

Is there any current research on as agents w/tools start dominating LLM use, if making making models smaller / less single-shot, more like efficient engines that can process a lot of context, and feeding a lot more into context windows is going to be more of a path forward vs trying to memory the world?

Like smaller models that show effectiveness on problems with verifiable rewards when run in a loop with external grounding context?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#318

Post by algoth1 »

libraryofbabel wrote: post #237

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the…

That’s basically the original gpt5 routing idea but done right

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#319

Post by furyofantares »

Klaus23 wrote: post #315
furyofantares wrote: post #312

Earlier quoted context omitted.

Here is the line in vLLM's source code that determines if a draft token is accepted: accepted = draft_prob > 0 and target_prob / draft_prob >= uniform_prob It does have a branch that checks only token id equality, which is used if temperature is 0.

Good analysis. That's surprising. I always heard that the draft model doesn't affect the output in any way. It seems they do it like this to achieve faster generation. It would be interesting to investigate how this affects the output. Edit: I haven't gone through all the code, but they might do something like this: https://arxiv.org/abs/2211.17192 where a draft model is used and the output distribution is tweaked on…

I have convinced myself that it is in fact the same distribution, even if you don't get the same output on any given run. Pretty cool.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#320

Post by HarHarVeryFunny »

manas96 wrote: post #298
libraryofbabel wrote: post #237

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the…

so in essence is it trading memory for speed?

Seems more like trading FLOPs for speed.

If you are just generating as usual with the main model then you're sequentially generating A -> AB -> ABC.

If I'm understanding correctly, what speculative decoding is doing is first (= more FLOPs) using a different small/fast (but less accurate) model to generate this ABC (you hope) sequence, then use the main model to now verify it in parallel (A + AB + ABC in parallel) rather then generate it sequentially. Assuming you had the FLOPs available to really do this in parallel, then this parallel verification vs sequential generation is what gives you the speed up.

Post reply on HN