Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

191–200 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#191

Post by nilkn »

aleksiy123 wrote: post #66

I’m starting to think that googles strategy is a bit different then the other frontier providers. Focusing more on performance to compute efficiency over pure performance. And maybe that’s why Gemini is (seemingly) lagging behind? Other providers hitting capacity and hitting the limits subsidising their inference. Google strategy seems to be about scaling and distributing these models to their existing billions of us…

I don't view Gemini as falling behind. I actually view it as a somewhat distinct type of intelligence compared to the latest iterations of GPT5 and Claude. The latter are, increasingly, very focused on productivity and automation of work tasks. They're optimized for long, agentic, self-correcting reasoning loops. Gemini is very different: it feels to me like a much smarter baseline model, with much deeper intuition (especially its Deep Think mode), but it's not nearly as good at long-range self-corrective agentic loops. For months now my workflow has been to use Gemini for creative leaps and insights, while preferring Codex or Claude or GPT5.5 Pro for routine or precision work.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#192

Post by lucb1e »

vasco wrote: post #157
lucb1e wrote: post #125

Earlier quoted context omitted.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for…

> in the best region (a few counties currently running on 100% hydropower) it makes no difference of course because the energy is running down the hill whether you use it or not. What? That's not how it works at all? Edit: dams release water when you need power or when they are full, not all the time

(It's past the edit/deletion window for my other comment, so placing a new one to reply to the edit)

Sure, but they're not infinitely large. I realized that it would be more accurate to mention this and edited that into the sentence after the one you quoted (you probably saw only the earlier version -- fair enough!), but either way, the average power consumption needs to be above the average water flow for it to not be 'wasted' (when the electric dam is already there anyway) so that part is basically free energy which we might as well use

Like, when electricity prices are negative in my area, I'm charging my EV (albeit a tiny one) no matter if I'm planning to drive tomorrow because there is a surplus anyhow and there might not be one when I want to charge next. Even without dynamic pricing, it costs me the same 35ct/kWh but there's just no reason not to, that I know of, until demand exceeds supply again. Even if they never shut down the coal plants (even during the heart of summer) and some of my electrons will be from coal, afaik every additional Wh used will come from the renewables rather than (like at night when the renewables have a fixed maximum supply) from the coal/gas plants. We don't have enough hydro storage around here to store even a single night's supply

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#193

Post by ritonlajoie »

basch wrote: post #117
XYen0n wrote: post #63

Earlier quoted context omitted.

The only thing a model can output is tokens; to achieve this, a tool of converting tokens into operational transformations is required. For example, I have an ast-grep skill, it will instruct the model to generate ast-grep rules and run ast-grep to perform file modifications.

I am saying to directly output the operational transformation instructions as the tokens. You’re essentially telling it to “write the diff” and then applying the patch. [retain(8), delete(6), insert("very very"), retain(10)]

there is a model in openrouter doing exactly this, it generates diffs. forgot the name though

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#195

Post by furyofantares »

petu wrote: post #128
furyofantares wrote: post #102

Earlier quoted context omitted.

> If the guess is right This is the crux. What makes the guess "right"? I think the acceptance criteria is not that the token is exactly the token the big model would have produced. It's accepted of the big model verifies that the probability of that token was high enough. How close it is to the same output (or same distribution of outputs) you'd get from running the big model would be dependent on temperature, top-k…

> What makes the guess "right"? Matching token that would've been picked without speculative decoding. That seems to be more or less agreed upon. e.g. vLLM docs list tests they run to ensure that output doesn't change if spec. decoding is used: https://github.com/vllm-project/vllm/blob/main/docs/features... But introducing some threshold to accept other high probability tokens is interesting idea.

By "lossless" I believe they mean "stays within the target distribution". Thats what their validation test says it tests. Maybe that means there is no loss in quality in practice. I don't think it means there is no change in output.

The paper they link to in that first paragraph says you compare logits to accept or reject.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#197

Post by nullc »

zdw wrote: post #9

MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.

Thanks for the link,it took qwen3.6-27B-q8 w/256k context on my RTX A6000 from ~20t/s to 55t/s. Prefill is mysteriously slower however, but prefill is so much faster still that I think I'm still bottlenecked on output most of the time.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#198

Post by ac29 »

brcmthrowaway wrote: post #194
ThouYS wrote: post #170

don't know about this guy, but qwen3.6:27b with the UD 4bit quant and little-coder/pi has been amazing. the first local LLM experience that can do actual meaningful work

What is UD?

Unsloth Dynamic, just some branding from Unsloth for their quants (other people use similar techniques)

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#199

Post by magicalhippo »

apexalpha wrote: post #114
nzeid wrote: post #91

Earlier quoted context omitted.

A few days ago I switched again from Qwen3.6 to Gemma 4 - for personal use I've experienced better average performance with the 26B version of the latter than the 27B of the former. For someone who's been running local models for a long while, these are very very exciting times.

I’ve been swapping between these too as well. However I find qwen unbeatable for toolcallling. I think gemma wasnt trained on that at all.

> However I find qwen unbeatable for toolcallling. I think gemma wasnt trained on that at all.

Gemma4 chat template seems to had multiple issues, at least with llama.cpp, not sure they're all fixed yet. It assumed simple types for parameters for example.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#200

Post by asdfasgasdgasdg »

Zarathruster wrote: post #68
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

Where are you using it? Is Gemini CLI at a usable state? It was a frustrating, miserable experience last time I gave it a shot. Antigravity seems significantly better in comparison, but with lower usage limits. If I run out, I usually don't bother switching to Gemini CLI.

I'm using it in antigravity, and fint it quite good. I have not managed to run out of usage on Flash. You can run Pro out of quota almost instantly, they really don't want you to use it if you're not paying $200 a month.

I do not use super broad prompts, though. None of this "build me a webapp" stuff. It's more like, "adjust this part of this class to do Y instead of X."

Post reply on HN