Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

261–270 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#261

Post by m12k »

libraryofbabel wrote: post #237

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the…

So we've basically taken the concept of branch prediction from CPUs and applied it to LLMs?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#262

Post by fragmede »

m12k wrote: post #261
libraryofbabel wrote: post #237

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the…

So we've basically taken the concept of branch prediction from CPUs and applied it to LLMs?

Well, the TPUs they're running on don't have branch prediction, so that had to end up somewhere in the stack.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#263

Post by cma »

zozbot234 wrote: post #248
mungoman2 wrote: post #244

Earlier quoted context omitted.

Naively it seems odd that running multiple checks in parallel is faster than just running the autoregressive model multiple times in series. It’s the same amount of compute right? But I think the key is that in the standard autoregressive case we get memory bandwidth bound, so there are tons of idle compute resources. And so checking multiple tokens is cheap because we can batch and thus reuse the read weights for mu…

> But I think the key is that in the standard autoregressive case we get memory bandwidth bound, so there are tons of idle compute resources. Right, this is the same way batching works. It's "free" until we exhaust available compute resources, at which point decode throughput becomes compute bound. (This is a good place to be, because scaling out compute is a lot easier than adding fast VRAM.) This is why MTP is most…

I think it also gets use in the /fast modes the providers sell at higher cost.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#264

Post by c7b »

m12k wrote: post #261
libraryofbabel wrote: post #237

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the…

So we've basically taken the concept of branch prediction from CPUs and applied it to LLMs?

The concept of predicting future elements in a series is not specific to CS. It's older than computers.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#265

Post by amdivia »

great_psy wrote: post #253

This might be silly, but … since the assistant models are so much smaller than the full models. What if we just use those smaller models? Any idea how much worse they will be ? Or is the issue that their error will really diverge as you accept more of their tokens?

I think they'll be extremely worse on their own

Predicting "America" in "The United States of ..." Is a different task from predicting the whole sentence.

So the small model is laying the blocks, and the bigger model would be cementing them in place or kicking them down. The bigger model's course correction is what keeps the smaller models predictions relatively on track

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#266

Post by seanhunter »

unethical_ban wrote: post #226
UncleOxidant wrote: post #220

Earlier quoted context omitted.

In the past I've usually found that Gemini (pro, flash) would get stuck on a problem and then seemingly start to do some kind of random search trying this and that just burning through tokens. When this would happen I'd switch (in antigravity) to Claude sonnet 4.6 and it would cut right to the chase and find the problem quickly. But the other day I was out of Claude tokens so I went back to Gemini 3.1 Pro and asked a…

Pardon my lack of depth on TFA here but in my experience with work, Gemini is far less accurate on queries about technical commands that Claude or OpenAI. Like, I don't trust it at all. Maybe it has its place but not as a general advisor.

I think what you’re seeing here is a difference in the amount of “world knowledge “ encoded in the perceptron parts of the model as opposed to how good the model is at the “transformer” part which you could think of as pure token prediction using only what’s in the context window.

If true that would suggest gemini/gemma would be great in a RAG situation where world model isn’t needed as it’s being spoonfed all the relevant information and less good at green field tasks.

That’s interesting to me because I have been struggling to understand how gemma4 is so good in my local use and how notebookLM does such a great job does when I give it project docs and yet gemini has always seemed behind claude when I use it cold for stuff.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#267

Post by zmmmmm »

libraryofbabel wrote: post #237

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the…

> it's fast to check that they are actually correct with the main model because you can run the checks in parallel.

Can you give an intuition as to why it's faster? I would have thought regardless how many you run in parallel, the successful check has to execute the full model to generate the full sequence so you will have exactly the same time needed? Or is it by process of elimination so it terminates early once it eliminates the non-viable choices? (in which case, how do you guarantee the correct output was speculatively generated at all to be the last survivor?)

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#268

Post by msp26 »

srigi wrote: post #144
msp26 wrote: post #43

Google is singlehandedly carrying western open source models. Gemma 4 31B is fantastic. However, it is a little painful to try to fit the best possible version into 24GB vram with vision + this drafter soon. My build doesn't support any more GPUs and I believe I would want another 4090 (overpriced) for best performance or otherwise just replace it altogether.

You could keep multimodal projector (understanding of audio, images & PDFs) in system RAM with `--no-mmproj-offload` in llama.cpp. Of course, then it is not accelerated with GPU, but you save its VRAM.

Interesting, I might try that, thanks!

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#270

Post by fulafel »

zmmmmm wrote: post #267
libraryofbabel wrote: post #237

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the…

> it's fast to check that they are actually correct with the main model because you can run the checks in parallel. Can you give an intuition as to why it's faster? I would have thought regardless how many you run in parallel, the successful check has to execute the full model to generate the full sequence so you will have exactly the same time needed? Or is it by process of elimination so it terminates early once it…

AIUI you run the checks of several predicted tokens in lockstep, and the computation for each token is served by the same data loaded from memory. In normal execution, each token would depend on the previous one, precluding the parallelization and causing much more per-token memory traffic.

So this is a case of trading off idle compute capacity that's waiting for the bottleneck (memory access).

Post reply on HN