Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

201–210 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#201

Post by 8note »

kissickas wrote: post #113
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

I only see plans for $8, $20, and $250/month... which one are you using exactly? https://gemini.google/subscriptions/

ive got the one that came with my phone.

its gotten much better on token limits and up time.

i recently reran a screenshot heavy task that i had last run in january, and it was able to keep running overnight and maybe peaked at 40% quota at any time, vs last time id need to resume it maybe twice to get the task to completion

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#202

Post by Craighead »

SwellJoe wrote: post #133
zobzu wrote: post #121

Earlier quoted context omitted.

gemma is also just way faster. i dont wanna wait 10min to get a 5-10% better answer (and sometimes, actually worse answer). best is to use your own model router atm, depending on the task

I'm pretty sure Qwen is faster? The MoE version of Qwen is 3B active, while Gemma 4 is 4B active. Similarly, the dense Qwen is 27B while Gemma is 31B. All else being equal (though I know all else isn't equal), Qwen should be faster in both cases. I haven't actually measured with any precision, but on my AMD hardware (Strix Halo or dual Radeon Pro V620) they seem quite similar in both cases...both MoE models are fast…

I'm using both incessantly and having a great time.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#203

Post by Schiendelman »

MengerSponge wrote: post #70
Urahandystar wrote: post #60

Earlier quoted context omitted.

True, but you have to add up the cumulative token output if your being fair. That alignment issue requires another set of input and output tokens to correct.

Does it? Or is this a centaur situation where a competent human can fix it in about two minutes?

Define competent. This is the difference between having a product manager able to prototype and having a product manager need to work with an engineer.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#204

Post by girvo »

nzeid wrote: post #91
zdw wrote: post #9

MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.

A few days ago I switched again from Qwen3.6 to Gemma 4 - for personal use I've experienced better average performance with the 26B version of the latter than the 27B of the former. For someone who's been running local models for a long while, these are very very exciting times.

Oh that's fascinating. 3.6 27B is pretty damned good, but slow in wall-clock times on my DGX Spark-alike. It generates huge reams of thinking before it gets the (usually correct!) answer, so wall-clock time is rough for tasks even at ~20tk/s

I'm surprised the 26B-A4B is better? It should be faster too, interesting. I'm excited to try 31B with MTP, because MTP-2 is what makes 27B bearable on the GB10.

What are you using it for? Agent-based coding, or something else?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#205

Post by [deleted] »

el_isma wrote: post #96

How is this different from the speculative decoding that we had before? You could pair a big and small model like qwen 32b with qwen 4b and had that same dynamic of the small model generating tokens and the big one "certifiying" them. The blog says something about re-using the big model's data?

[deleted]

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#206

Post by onlyrealcuzzo »

rjh29 wrote: post #59
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

I find Gemini to be quite good / acceptable at code review, design, and design review, but it's notably far behind Claude Code for implementation.

Are you having better results?

Codex is fast and decent, but I REALLY have to stay on top of it. The amount of times it makes executive design decisions on the fly to completely break everything is way too high.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#207

Post by [deleted] »

rjh29 wrote: post #59
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

[deleted]

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#208

Post by basiccalendar74 »

furyofantares wrote: post #102
Klaus23 wrote: post #94

Earlier quoted context omitted.

It really is. This is because LLMs with a single output/user are strongly bandwidth limited. Although the hardware can generate multiple tokens simultaneously, it is slowed down if the tokens depend on each other, as is the case with regular text generation. The draft model essentially predicts the next token quickly, enabling you to start generating the subsequent token in parallel. If the guess is right, the second…

> If the guess is right This is the crux. What makes the guess "right"? I think the acceptance criteria is not that the token is exactly the token the big model would have produced. It's accepted of the big model verifies that the probability of that token was high enough. How close it is to the same output (or same distribution of outputs) you'd get from running the big model would be dependent on temperature, top-k…

it is only "right" statistically as in conforming to the same distribution. but there is no guarantee of exact same output.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#209

Post by bikelang »

gavmor wrote: post #109
macNchz wrote: post #25

Earlier quoted context omitted.

This is something I've been thinking about for a while...the current state of things really does feel kind of like the dialup era, wondering what the "broadband" era could look like. Watching tokens stream in is reminiscent of watching a jpeg load a few rows of pixels at a time, and the various different loading and connecting animations that applications implemented before things got fast enough to make them less re…

Take a look at https://chatjimmy.ai/ -- it's running against Taalas' "hardcore" silicon model, ie a dedicated, ASIC-like chip.

Wow - actually pretty astonishing how fast their inference is. So fast it feels fake?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#210

Post by vparseval »

SwellJoe wrote: post #133
zobzu wrote: post #121

Earlier quoted context omitted.

gemma is also just way faster. i dont wanna wait 10min to get a 5-10% better answer (and sometimes, actually worse answer). best is to use your own model router atm, depending on the task

I'm pretty sure Qwen is faster? The MoE version of Qwen is 3B active, while Gemma 4 is 4B active. Similarly, the dense Qwen is 27B while Gemma is 31B. All else being equal (though I know all else isn't equal), Qwen should be faster in both cases. I haven't actually measured with any precision, but on my AMD hardware (Strix Halo or dual Radeon Pro V620) they seem quite similar in both cases...both MoE models are fast…

qwen-3.6 is really interesting. The dense 27B model is pretty slow for me whereas the sparse 31B is blazingly fast but it also needs to be since it's so chatty. It produces pages and pages of stream of consciousness stuff. 27B does this to a lesser extent but slow enough that I can actually read it whereas 31B just blasts by.

I haven't yet compared either to Gemma 4. I tried that out the day after it came out with the patched llama.cpp that added support for it but I couldn't make tool calling work and so it was kind of useless. I should try again to see if things have changed but judging by what people say, qwen-3.6 seems stronger for coding anyway.

Post reply on HN