Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

181–190 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#181

Post by jalcazar »

Zarathruster wrote: post #68
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

Where are you using it? Is Gemini CLI at a usable state? It was a frustrating, miserable experience last time I gave it a shot. Antigravity seems significantly better in comparison, but with lower usage limits. If I run out, I usually don't bother switching to Gemini CLI.

I tried it the very first day it was available to Google employees, and it was not usable.

Then a few weeks back, I gave it another try and I was pleasantly surprised.

It was insanely good!

A colleague and I have been on-and-off trying to build a C++ binary against specific Google libraries for months without success. Then, Gemini CLI was able to build the binary after 2-3 days iterating and refining prompts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#182

Post by xnx »

kissickas wrote: post #113
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

I only see plans for $8, $20, and $250/month... which one are you using exactly? https://gemini.google/subscriptions/

The Google One plans are also good deals: https://one.google.com/about/google-ai-plans/

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#183

Post by mchusma »

nolist_policy wrote: post #38
mchusma wrote: post #2

I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?

What do you mean? It just works with Google AI Studio.

Part of the issue is Google complex web of products. There’s vertex Gemini Google AI studio Google edge. But I literally had trouble finding how to use this in my existing paid Gemini API account.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#184

Post by mchusma »

Farmadupe wrote: post #5
mchusma wrote: post #2

I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?

I wonder if for a model that small with a permissive license it might not be worth their time to host a commercial grade inference stack? Might be easier to chuck it over the fence and let other providers handle it as it'll run in almost any commercial grade card? Also speculating, but I wonder if it might also create a bit of a pricing problem relative to Gemini flashlight depending on serving cost and quality of ou…

If you were to believe a lot of metrics Gemma 31B it’s much better than flash lite. It seems like I should be able to pay Google to use it and that should be at least a secretary, called action how I can do that but it’s missing from both the blog post entirely.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#185

Post by egeres »

zdw wrote: post #9

MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.

There's also a growing interest on integrating DFlash: https://github.com/ggml-org/llama.cpp/issues/21978, I can't wait to see how it will compare against MTP

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#186

Post by furyofantares »

Klaus23 wrote: post #143
furyofantares wrote: post #102

Earlier quoted context omitted.

> If the guess is right This is the crux. What makes the guess "right"? I think the acceptance criteria is not that the token is exactly the token the big model would have produced. It's accepted of the big model verifies that the probability of that token was high enough. How close it is to the same output (or same distribution of outputs) you'd get from running the big model would be dependent on temperature, top-k…

The token is correct if it matches the one generated by the main model. It works like this: The draft model quickly generates draft-token 1. The main model then starts working on two tokens in parallel. It calculates token 1 based on the context, and token 2 based on the context + draft-token 1. Once the two tokens have been generated, you can check whether the draft-token 1 from the draft model matches token 1 from…

Models do not generate tokens. They generate probabilities for each token.

Inference parameters select a token using those.

You can just select the top token all the time or you can do it probabilistically.

How you do that in both the speculative decoding and the main inference changes how likely you get the exact same tokens. And then you can choose to accept only if the token matches exactly, or you can choose to accept if it was reasonably likely to be chosen.

Let's say the main model picked the 2nd most likely token and speculative picked the most likely. You can reject that - but you get less speed up. You can accept it, you get more speed up, but you do change the output. You risk the distribution of your outputs not being what you hope.

I am simplifying. I know in https://arxiv.org/pdf/2302.01318 they specify a probability that you reject a token.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#187

Post by TOMDM »

Lihh27 wrote: post #105
vhiremath4 wrote: post #51

So this is like branch prediction for operating systems? Except we have probability baked into the model itself so it’s even more reliable.

similar idea, but the failure mode is better. a branch mispredict burns cycles. a bad guess here usually just means no bonus tokens. https://arxiv.org/abs/2211.17192

As long as you're not bound on parallelism or bandwidth then it's "free", but if you're constrained on either resource then your lighter predictor model just needs to save you more cycles than it congests on average.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#188

Post by 332451b »

lucb1e wrote: post #146
losteric wrote: post #132

Earlier quoted context omitted.

> using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, How do you get to this range? That's quite a spread. When I last ran the math, my daily usage (efficient and effective productivity, not spamming Gas Town) came to about 0.67 kg of CO2, which is roughly equivalent to my individual emissions from the 1 mile public bus ride home from work.

Data is from https://app.electricitymaps.com/map The difference is so big because renewables are just that much more efficient than coal and, to a lesser extent, natural gas. You can have 60% coming from renewable sources and still emit 400g/kWh with a coal and gas mix, whereas all hydro is 24g/kWh according to that source. The production component is what makes renewables not completely emission-free

I'm curious how you got the energy consumption for 8 hours of use.

I would imagine there's a huge spread there too. Depending on hardware, size of the model, requests being batched and queued rather than occupying a whole GPU or TPU, or if you are running an agent loop for 8 hours straight or spend most of the time reviewing results.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#189

Post by endymi0n »

zdw wrote: post #9

MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.

I don’t exactly know where MTP inference fits within the inference stack, but does someone know whether it’s possible to implement it for the MLX universe?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#190

Post by mnicky »

WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

In the Dwarkesh's podcast Dylan Patel from SemiAnalysis said that Google can currently afford to have larger models than competitors, because of access to much more compute, TPUs etc.

That could explain the token usage difference because larger models usually use less tokens per the same unit of intelligence.

Post reply on HN