Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

171–180 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#171

Post by wrxd »

gunalx wrote: post #168
wrxd wrote: post #167

I'm not sure I understand how this work https://huggingface.co/google/gemma-4-E4B-it-assistant has 78.8M parameters while the standard variant https://huggingface.co/google/gemma-4-E4B-it has 8B parameters. Is gemma-4-E4B-it-assistant a model I can use stand-alone or a model I need to use in combination with gemma-4-E4B-it?

You need the regular gemma model as well. You can think of this as a really small distillation of the original. Useless by its own because it often is wrong, but it is fifth more than not. And because verifying a transformer model can be done faster than running it. We can effectively speed up by using this draft model and only doing the compute where it was wrong. This is a oversimplification, but tldr you need both…

Thank you!

I already played with Gemma4 on oMLX a while ago. When I have some time I'll check if it supports running MTP models and play a bit more

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#172

Post by Sabinus »

kissickas wrote: post #113
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

I only see plans for $8, $20, and $250/month... which one are you using exactly? https://gemini.google/subscriptions/

At least the $20 one. The $8 plan has the same cli limits as an unpaid account.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#173

Post by mcv »

WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

One of the consequences of Gemma's speed is that you can run it on a GPU that's technically too small for it. I've run it on my 4070, and while the output wasn't blazingly fast, it was usable. (Though I haven't used it for anything complex yet. I'm sure that will be different.)

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#174

Post by nullsanity »

rjh29 wrote: post #59
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

[dead]

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#175

Post by diordiderot »

rjh29 wrote: post #59
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

I find it really really slow compared to gpt/Claude

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#176

Post by kingleopold »

kissickas wrote: post #115
kingleopold wrote: post #88

Earlier quoted context omitted.

no 15/month does not enough all day? pls dont share wrong info, 3.1 pro CLI sometimes wait 20-30 min thinking sometimes, it's by far worse compared to others.It finishes with few hours of work mostly, but in openai they give you 6 times of that in 24 hours, gemini resets one time a day. It is literally lazy and so many times does half work. I'm a power user for all top models in top 3 AI companies, only Gemini 3.1 wa…

Which do you find best? I am using Claude Code but hit the 5-hour limits easily, and burn through the weekly allowance in 3-4 days... and I'm not even using it for work

gpt 5.5 is really good, CC is really expensive but it's similar level.

Gemini 3.1 and 3 flash are only good for more simple tasks and when work is not the important part of the project

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#177

Post by maille »

jszymborski wrote: post #29
christina97 wrote: post #28

I recently set up the 26B A4B model up on vLLM on an RTX3090 (4-bit) after a hiatus from local models. Just completely blown away by the speed and quality you can get now for sub-$1k investment. I tried first with Qwen but it was unstable and had ridiculously long thinning traces!

The A4B model is blazing fast and the model is super good at general inquiries. Notably worse than Qwen 3.6 for coding tasks but that says more about the Qwen model.

Bad at coding, but would it be good at code review?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#178

Post by dbreunig »

WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Among benchmarkers its a frequent topic. Qwen BURNS reasoning to get its scores.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#179

Post by sureMan6 »

mapontosevenths wrote: post #166
zobzu wrote: post #116

Earlier quoted context omitted.

flash is the fast (duh) model though. its not always beneficial to use pro. in practice: 1/ set to flash 3.1 ; 2/ force to pro...sometimes. mainly when the cli fails to predict what model to use. note that it will sometimes fall back to flash 2, which sucks

Flash will absolutely destroy a complex codebase. It's like a drunk junior programmer. Don't trust it with anything more complex than autocomplete. Pro is expensive, but good. However they've decreased the pitiful stipend they used to include in even the ultra plan to the point were it's barely usable. I pivoted back to ChatGPT Pro after the recent downgrade they gave Ultra users. Googles Ultra plan cost 2.5x as much…

Yeah I don't get the user who said Gemini is generous with the quota, I get more use out of codex with the 5 hour limits than Gemini gives me in a week

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#180

Post by xnx »

WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Claude is very fashionable right now, but I've never had any problems or felt the need to switch.

Maybe after Google I/O, more people will catch on to how good it is.

Post reply on HN