Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

291–300 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#291

Post by sigmoid10 »

basch wrote: post #141
sigmoid10 wrote: post #131

Earlier quoted context omitted.

Sounds easy, but isn't in practice. You can look at the edit text file tool in va code copilot for example to see how complicated that can get: https://github.com/microsoft/vscode-copilot-chat/tree/9e668c...

I have no idea when I’m being lied to anymore but allegedly Aider and Cursor work the way I described, although cursor is using a second model to apply the edit.

They all do something similar under the hood. Patching files is not a trivial task when you only have the changed text content and not the actual file structure to work with. It kind of works, but is fundamentally limited by the LLM output architecture.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#292

Post by lossolo »

mike_hearn wrote: post #275
m12k wrote: post #261

Earlier quoted context omitted.

So we've basically taken the concept of branch prediction from CPUs and applied it to LLMs?

Maybe at very high level of abstraction, but there's no branching involved.

Well, there are multiple token proposals processed in parallel, from which only one is picked, seems like branching to me. The only difference is that in case of CPU there is always only one possible branch that is correct.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#293

Post by esyir »

zmmmmm wrote: post #285
janalsncm wrote: post #282

Earlier quoted context omitted.

The small draft model proposes a sequence of tokens d1 d2 d3. The big target model calculates P(d1) P(d2|d1) P(d3|d1 d2) In parallel. If we were just greedy decoding it would be simple. Just stop when the draft model doesn’t predict the most likely token as judged by the target model. At that point, append the correct token from the target model and kick off both models again in parallel. In practice we aren’t using…

nice ... i think i get the idea - it's effectively the same / similar benefit as batching, but you're batching against your own speculated future path. Which would be pointless if you didn't have a high probability path to evaluate against - but the draft gives you that.

I'll add an expansion here. It's more useful to you locally, as you have excess compute that's generally wasted. If you're serving multiple user and trying to max output, you might cost some in this case

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#294

Post by kingleopold »

rjh29 wrote: post #258
kingleopold wrote: post #88

Earlier quoted context omitted.

no 15/month does not enough all day? pls dont share wrong info, 3.1 pro CLI sometimes wait 20-30 min thinking sometimes, it's by far worse compared to others.It finishes with few hours of work mostly, but in openai they give you 6 times of that in 24 hours, gemini resets one time a day. It is literally lazy and so many times does half work. I'm a power user for all top models in top 3 AI companies, only Gemini 3.1 wa…

"Wrong info" lol. We just have different use patterns or expectations. Saying you're a "AI power user" is not the appeal to authority you think it is. Everybody here is using AI.

great comment with lots of information in it, you best!

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#296

Post by fulafel »

adrian_b wrote: post #106
el_isma wrote: post #96

How is this different from the speculative decoding that we had before? You could pair a big and small model like qwen 32b with qwen 4b and had that same dynamic of the small model generating tokens and the big one "certifiying" them. The blog says something about re-using the big model's data?

Multi token prediction is the same thing as speculative decoding. This is mentioned in the Google pages describing their MTP implementation. Google has now provided small models for each of the previous Gemma 4 models, e.g. "gemma-4-26B-A4B-it-assistant" for "gemma-4-26B-A4B-it". The difference vs. Qwen is that here each small model is not some general-purpose smaller model, but a model that has been optimized specif…

Multi-token prediction is a refined form of speculative decoding.

Researchers at Google came up with Speculative decoding in 2022: https://research.google/blog/looking-back-at-speculative-dec... (Fast Inference from Transformers via Speculative Decoding - Yaniv Leviathan, Matan Kalman, Yossi Matias)

Researchers at Meta came up with MTP, a smarter way of doing speculative decoding in 2024: https://arxiv.org/abs/2404.19737 (Better & Faster Large Language Models via Multi-token Prediction Fabian Gloeckle, Badr Youbi Idrissi, Baptiste Rozière, David Lopez-Paz, Gabriel Synnaeve)

DeepSeek V3 shipped MTP in a product first, in 2024: https://arxiv.org/abs/2412.19437 (DeepSeek-V3 Technical Report, 100+ authors)

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#297

Post by kridsdale1 »

mike_hearn wrote: post #274
zmmmmm wrote: post #267

Earlier quoted context omitted.

> it's fast to check that they are actually correct with the main model because you can run the checks in parallel. Can you give an intuition as to why it's faster? I would have thought regardless how many you run in parallel, the successful check has to execute the full model to generate the full sequence so you will have exactly the same time needed? Or is it by process of elimination so it terminates early once it…

An obscure fact about the transformer architecture is that it more or less computes the most likely next token for every single token in the context window at once. This is because the KV cache values needed to predict the next token are needed for every token, and the attention modules do nearly all the work, so once you computed the KVs running them through the last sections to get the target probabilities is nearl…

As an EECS who is now in ML I think this post was well written. Thanks.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#298

Post by manas96 »

libraryofbabel wrote: post #237

Speculative decoding is an amazingly clever invention, almost seems-too-good-to-be-true (faster interference with zero degradation from the quality of the main model). The core idea is: if you can find a way to generate a small run of draft next tokens with a smaller model that have a reasonable likelihood of being correct, it's fast to check that they are actually correct with the main model because you can run the…

so in essence is it trading memory for speed?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#299

Post by kridsdale1 »

jalcazar wrote: post #181
Zarathruster wrote: post #68

Earlier quoted context omitted.

Where are you using it? Is Gemini CLI at a usable state? It was a frustrating, miserable experience last time I gave it a shot. Antigravity seems significantly better in comparison, but with lower usage limits. If I run out, I usually don't bother switching to Gemini CLI.

I tried it the very first day it was available to Google employees, and it was not usable. Then a few weeks back, I gave it another try and I was pleasantly surprised. It was insanely good! A colleague and I have been on-and-off trying to build a C++ binary against specific Google libraries for months without success. Then, Gemini CLI was able to build the binary after 2-3 days iterating and refining prompts

Hello fellow Googler. Please give Antigravity’s gLinux CLI a try. (That’s not its name but I won’t put an internal code name here, I hope you know what I mean).

I moved to it from Gemini CLI last week and it is phenomenally faster and more reliable. It only took about an hour to get all my hooks and skills ported.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#300

Post by kridsdale1 »

chrisweekly wrote: post #221
mapontosevenths wrote: post #166

Earlier quoted context omitted.

Flash will absolutely destroy a complex codebase. It's like a drunk junior programmer. Don't trust it with anything more complex than autocomplete. Pro is expensive, but good. However they've decreased the pitiful stipend they used to include in even the ultra plan to the point were it's barely usable. I pivoted back to ChatGPT Pro after the recent downgrade they gave Ultra users. Googles Ultra plan cost 2.5x as much…

Tangent: this is one of those situations where slang is harmful to understanding. When I saw "will absolutely destroy" my first interpretation was a positive connotation. Of course further context made it clear you were being straightforward, and this isn't aimed at you. Along these lines, "drop" has become a problematic term: "Acme co dropped support for Foo" means it's EOL, but "Foo dropped today" implies it just l…

I had the same experience.
Post reply on HN