Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

151–160 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#151

Post by throwaway219450 »

aleksiy123 wrote: post #118
leecommamichael wrote: post #81

Earlier quoted context omitted.

Isn't that where everyone's strategy is shifting?

Yes, but I think Google was playing that strategy from essentially day 1 or very early in this AI race, where as the others are there now because of their lack of access of compute. The general narrative I would read on HN/others, was that Google would be able to outlast/outcompete OpenAI and Anthropic because Google had both more money and more compute. Playing the game of subsidizing their most capable models to ca…

It benefits Google's bottom line to have very capable small models that can cheaply cache results for search queries, even if they're frequently wrong. But I wonder if they use Gemini for the top X% of search terms to try and get better retention? Also the TPU vertical gives a good advantage here. I've never been super impressed with Gemini out of the box, but surely, surely, Google is best positioned here.

As a consumer, 24-32 GB VRAM is affordable ($1-2 k) and that's the frontier I'm most interested in. It's very "two papers down the line". Those models are also feasible to fine-tune, unlike the O(100+B) behemoths. The 4000 Pro Blackwell has very good TDP compared to people insisting on using 300-600W gaming cards. If I was freelancing, I would definitely consider getting a 6000 for work.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#152

Post by moffkalast »

pu_pe wrote: post #24

So much faster inference with no quality degradation? All that for just some small memory overhead (drafter models are <1B it seems)?

It's based on taking advantage of spare compute if you have it. A tiny model generates a few steps ahead first, then the large one runs batch inference on all of those at once as if you are at that point in time. If they all check out afterwards it jumps ahead, otherwise it discards and goes onto the next one.

Not sure about this implementation, but conceptually it only works well on very capable GPUs for very predictable output. Typical speedup is about 30%, not sure how google is claiming 250% which is ridiculous.

And if you don't have enough compute, then you get negative speedup from all the extra overhead.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#153

Post by nolist_policy »

sigmar wrote: post #61

>try them directly on Google AI Edge Gallery for Android or iOS. I'm not seeing any update to the app on my android phone... maybe later today? >We’ve published an in-depth technical explainer I was expected a pdf link, but this goes to a brief article on twitter/X. lol, okay...

It's up on GitHub: https://github.com/google-ai-edge/gallery/releases

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#156

Post by [deleted] »

beavisringdin wrote: post #67
zargon wrote: post #47

Earlier quoted context omitted.

Groq.

No, it was a custom ASIC chip with weights baked in for a singular model. I do envision a future where we return to cartridges. Local AI is de facto and massively optimised chips are built to be plug and play running a single SoTA model.

[deleted]

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#157

Post by vasco »

lucb1e wrote: post #125
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for…

> in the best region (a few counties currently running on 100% hydropower) it makes no difference of course because the energy is running down the hill whether you use it or not.

What? That's not how it works at all?

Edit: dams release water when you need power or when they are full, not all the time

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#158

Post by lucb1e »

vasco wrote: post #157
lucb1e wrote: post #125

Earlier quoted context omitted.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for…

> in the best region (a few counties currently running on 100% hydropower) it makes no difference of course because the energy is running down the hill whether you use it or not. What? That's not how it works at all? Edit: dams release water when you need power or when they are full, not all the time

Do explain!

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#159

Post by lucb1e »

in-silico wrote: post #139
lucb1e wrote: post #125

Earlier quoted context omitted.

I don't know if people know this, but using it all day (say 8h) costs between 0.7 and about 14 kg of CO2 in the US, depending on which region's grid power they use (or, if they run off of generators, the gCO2e/kWh might be very different from these bounds). With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for…

Using the logarithmic mean of your range of about 3 kg of CO2 per day, and the fact that the average car emits about 0.2 kg of CO2 per km, this means that a typical day of Gemini coding produces about the same amount of CO2 as a 15 km (~9 mile) round-trip commute by car.

You can't average it like that because it's not an evenly random distribution. (And a place has to be very high in renewables, like on the order of 95%, before the emissions aren't dominated by the fossil component.) I don't know what the average datacenter uses for electricity source or region

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#160

Post by walthamstow »

Zarathruster wrote: post #68
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

Where are you using it? Is Gemini CLI at a usable state? It was a frustrating, miserable experience last time I gave it a shot. Antigravity seems significantly better in comparison, but with lower usage limits. If I run out, I usually don't bother switching to Gemini CLI.

It's definitely not as good as Codex or Claude Code but it is cheap. You just have to manage it a bit more. I got a year for free with my phone and I still pay for Codex, so take from that what you will.
Post reply on HN