Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

211–220 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#211

Post by rapind »

rjh29 wrote: post #59
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

Just a heads up that you cannot opt out of training on any of their "personal" plans (including Ultra) last time I checked. Both Claude and ChatGPT allow you to opt out of training on their paid plans.

It would be nice if this was a bit more obvious and clear too.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#212

Post by _factor »

nullc wrote: post #197
zdw wrote: post #9

MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.

Thanks for the link,it took qwen3.6-27B-q8 w/256k context on my RTX A6000 from ~20t/s to 55t/s. Prefill is mysteriously slower however, but prefill is so much faster still that I think I'm still bottlenecked on output most of the time.

Took 2x AMD MI50s to 50 t/s instead of 20 t/s for Q8 27B. Impressive.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#213

Post by tjwebbnorfolk »

lucb1e wrote: post #125
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for…

How much CO2 did your computer burn while you wrote such a long and pointless comment

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#215

Post by qingcharles »

asdfasgasdgasdg wrote: post #200
Zarathruster wrote: post #68

Earlier quoted context omitted.

Where are you using it? Is Gemini CLI at a usable state? It was a frustrating, miserable experience last time I gave it a shot. Antigravity seems significantly better in comparison, but with lower usage limits. If I run out, I usually don't bother switching to Gemini CLI.

I'm using it in antigravity, and fint it quite good. I have not managed to run out of usage on Flash. You can run Pro out of quota almost instantly, they really don't want you to use it if you're not paying $200 a month. I do not use super broad prompts, though. None of this "build me a webapp" stuff. It's more like, "adjust this part of this class to do Y instead of X."

Also bonus: using it in Antigravity you can burn through all the Opus credit Google give you first to do all the planning and then switch it to Gemini 3.1 Pro to do the grunt work.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#216

Post by qingcharles »

bikelang wrote: post #209
gavmor wrote: post #109

Earlier quoted context omitted.

Take a look at https://chatjimmy.ai/ -- it's running against Taalas' "hardcore" silicon model, ie a dedicated, ASIC-like chip.

Wow - actually pretty astonishing how fast their inference is. So fast it feels fake?

Yeah, when you find fast inference like that it almost feels like the answer arrives before you hit return. Now imagine it running locally with no server round-trip.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#217

Post by glenngillen »

nzeid wrote: post #91
zdw wrote: post #9

MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.

A few days ago I switched again from Qwen3.6 to Gemma 4 - for personal use I've experienced better average performance with the 26B version of the latter than the 27B of the former. For someone who's been running local models for a long while, these are very very exciting times.

I've been thinking about doing more of this too. What spec machine are you running? And are you using long-running autonomous agents or more of the IDE/co-pilot style of collaboration?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#218

Post by saintfire »

divan wrote: post #134
lucb1e wrote: post #125

Earlier quoted context omitted.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for…

Normal human exhales roughly 0.7-1.0 kg of CO2 over 8h.

And how much do they exhale over 8h of AI use?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#219

Post by fomoz »

mchusma wrote: post #2

I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?

You can use it for free with Google AI studio (free tier or paid tier accounts with different limits). Or use the paid version from Vertex AI which is around 3x cheaper than Gemini 3 Flash.

I'm using Gemma 4 31B in my app with 5 agents, 1.5k requests per day, each.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#220

Post by UncleOxidant »

rjh29 wrote: post #59
WarmWash wrote: post #56

I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

In the past I've usually found that Gemini (pro, flash) would get stuck on a problem and then seemingly start to do some kind of random search trying this and that just burning through tokens. When this would happen I'd switch (in antigravity) to Claude sonnet 4.6 and it would cut right to the chase and find the problem quickly. But the other day I was out of Claude tokens so I went back to Gemini 3.1 Pro and asked about a verilog simulation problem that Claude had been stuck on - and it figured it out in a few minutes.
Post reply on HN