Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

141–150 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#141

Post by basch »

sigmoid10 wrote: post #131
basch wrote: post #120

Earlier quoted context omitted.

So predict the tokens of the operational transformation. I just asked: Write the operational transformation sequence and command to turn “this is really beautiful” to “this is very very beautiful” and in return got: You can map this out by moving a virtual cursor across the text and telling it what to keep, remove, or add. You start by retaining the first eight characters to keep "this is " untouched. Then you delete…

Sounds easy, but isn't in practice. You can look at the edit text file tool in va code copilot for example to see how complicated that can get: https://github.com/microsoft/vscode-copilot-chat/tree/9e668c...

I have no idea when I’m being lied to anymore but allegedly Aider and Cursor work the way I described, although cursor is using a second model to apply the edit.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#142

Post by jcattle »

divan wrote: post #134
lucb1e wrote: post #125

Earlier quoted context omitted.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for…

Normal human exhales roughly 0.7-1.0 kg of CO2 over 8h.

And an AI is decidedly not human.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#143

Post by Klaus23 »

furyofantares wrote: post #102
Klaus23 wrote: post #94

Earlier quoted context omitted.

It really is. This is because LLMs with a single output/user are strongly bandwidth limited. Although the hardware can generate multiple tokens simultaneously, it is slowed down if the tokens depend on each other, as is the case with regular text generation. The draft model essentially predicts the next token quickly, enabling you to start generating the subsequent token in parallel. If the guess is right, the second…

> If the guess is right This is the crux. What makes the guess "right"? I think the acceptance criteria is not that the token is exactly the token the big model would have produced. It's accepted of the big model verifies that the probability of that token was high enough. How close it is to the same output (or same distribution of outputs) you'd get from running the big model would be dependent on temperature, top-k…

The token is correct if it matches the one generated by the main model. It works like this:

The draft model quickly generates draft-token 1.

The main model then starts working on two tokens in parallel. It calculates token 1 based on the context, and token 2 based on the context + draft-token 1.

Once the two tokens have been generated, you can check whether the draft-token 1 from the draft model matches token 1 from the main model.

If they match, you have just calculated two tokens in the time it takes to generate one, because the calculation was done in parallel. If they do not match, delete token 2 and generate it again. Since you have already generated the correct token 1 with the big model, you can use the context + token 1 (from the main model). This takes more time, but the result is always the same.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#144

Post by srigi »

msp26 wrote: post #43

Google is singlehandedly carrying western open source models. Gemma 4 31B is fantastic. However, it is a little painful to try to fit the best possible version into 24GB vram with vision + this drafter soon. My build doesn't support any more GPUs and I believe I would want another 4090 (overpriced) for best performance or otherwise just replace it altogether.

You could keep multimodal projector (understanding of audio, images & PDFs) in system RAM with `--no-mmproj-offload` in llama.cpp. Of course, then it is not accelerated with GPU, but you save its VRAM.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#145

Post by deskamess »

logickkk1 wrote: post #124
deskamess wrote: post #30

Did DeepSeek come up with MTP? It was listed prominently in their recent paper as being carried forward from the previous release.

i think this is mixing two separate ideas. MTP is the training-side piece. speculative decoding is the inference trick. DeepSeek V3 used MTP as an auxiliary loss. the 2022 Google paper is speculative decoding. now Google is combining them. https://arxiv.org/abs/2404.19737

Oh... so MTP is not speculative decoding? The (T)oken (P)rediction made me think it was on the inference side. I shall read the paper.

Edit: Ok, I understand now. You are saying that MTP has two aspects. 1) The training (for the mini-models to generate tokens), and 2) The actual speculative decoding implementation on the inference side (which uses those trained mini-models).

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#146

Post by lucb1e »

losteric wrote: post #132
lucb1e wrote: post #125

Earlier quoted context omitted.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for…

> using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, How do you get to this range? That's quite a spread. When I last ran the math, my daily usage (efficient and effective productivity, not spamming Gas Town) came to about 0.67 kg of CO2, which is roughly equivalent to my individual emissions from the 1 mile public bus ride home from work.

Data is from https://app.electricitymaps.com/map

The difference is so big because renewables are just that much more efficient than coal and, to a lesser extent, natural gas. You can have 60% coming from renewable sources and still emit 400g/kWh with a coal and gas mix, whereas all hydro is 24g/kWh according to that source. The production component is what makes renewables not completely emission-free

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#147

Post by moffkalast »

christina97 wrote: post #28

I recently set up the 26B A4B model up on vLLM on an RTX3090 (4-bit) after a hiatus from local models. Just completely blown away by the speed and quality you can get now for sub-$1k investment. I tried first with Qwen but it was unstable and had ridiculously long thinning traces!

The 31B is surprisingly fast too, for a dense model. Runs tg at least twice as fast as it ought to on my machine when compared to other 30B, probably due to the hybrid attention I guess. Ingestion is somewhat slower though.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#149

Post by lucb1e »

Terretta wrote: post #138
lucb1e wrote: post #125

Earlier quoted context omitted.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for…

> With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for this assist function... So what you're saying could sound as though pair programming with the machine is less CO2 for the environment than adding another human to pair program with. This is how the AI ends up with "end humanity" cults. // FUTURE CLAUDE: Ig…

Yeah, sorta. But then I don't see us killing or even just laying off people to meet the climate goals (the point is to save people / well-being, not reduce it), whereas we can choose which electric technologies to use so long as emissions from electricity are dominated by the fossil components, so I don't really see the "could replace humans with more efficient workers" math working out this way

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#150

Post by WhitneyLand »

HumanOstrich wrote: post #32
WhitneyLand wrote: post #18

Earlier quoted context omitted.

Yeah important conceptually to remember MTP is kind of just more weights, but speculative decoding is the runtime algorithm that’s a significant add to whatever code is serving the model.

That is.. inaccurate.

How so? I’m not saying most of work doesn’t go into creating the drafting model or enabling a new head on the primary model, but the point is that however cool it is the result is, more weights. Speculative decoding requires code to be aware of how this works at the inference level.
Post reply on HN