Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Accelerating Gemma 4: faster inference with multi-token prediction drafters

blog.google

Post reply on HN
Print view View original

221–230 of 345 posts

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#221

Post by chrisweekly »

mapontosevenths wrote: post #166
zobzu wrote: post #116

Earlier quoted context omitted.

flash is the fast (duh) model though. its not always beneficial to use pro. in practice: 1/ set to flash 3.1 ; 2/ force to pro...sometimes. mainly when the cli fails to predict what model to use. note that it will sometimes fall back to flash 2, which sucks

Flash will absolutely destroy a complex codebase. It's like a drunk junior programmer. Don't trust it with anything more complex than autocomplete. Pro is expensive, but good. However they've decreased the pitiful stipend they used to include in even the ultra plan to the point were it's barely usable. I pivoted back to ChatGPT Pro after the recent downgrade they gave Ultra users. Googles Ultra plan cost 2.5x as much…

Tangent: this is one of those situations where slang is harmful to understanding. When I saw "will absolutely destroy" my first interpretation was a positive connotation. Of course further context made it clear you were being straightforward, and this isn't aimed at you. Along these lines, "drop" has become a problematic term: "Acme co dropped support for Foo" means it's EOL, but "Foo dropped today" implies it just landed. Idioms are hard enough when they don't serve as borderline autoantonyms. To wrap up this extended digression, if anyone else finds this sort of thing interesting, and could use a good laugh, check out Ismo (a standup comic from Finland who makes truly hilarious observations about English as a second language).

https://youtu.be/oGmzfjuicE0?si=nL_W75s8UDp1g-zI

https://youtu.be/jXcMoHeWaYQ?si=QMi7nEwVWvCZyzbl

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#222

Post by snek_case »

lelandbatey wrote: post #100
jeffhuys wrote: post #39

Earlier quoted context omitted.

Check chatjimmy.ai

https://chatjimmy.ai being a demo of the "burn the model to an ASIC" approach being sold by Taalas[0], an approach which they use to run Llama 3.1 8B at ~17000 tokens per second. [0] - https://taalas.com/products/

Not to downplay their accomplishment but Llama 3.1 8B is a terrible model. It's really outdated at this point. It's cool that they were able to accelerate a model with silicon, but it also feels wasteful since llama 8B is such a useless model?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#223

Post by xnx »

qingcharles wrote: post #215
asdfasgasdgasdg wrote: post #200

Earlier quoted context omitted.

I'm using it in antigravity, and fint it quite good. I have not managed to run out of usage on Flash. You can run Pro out of quota almost instantly, they really don't want you to use it if you're not paying $200 a month. I do not use super broad prompts, though. None of this "build me a webapp" stuff. It's more like, "adjust this part of this class to do Y instead of X."

Also bonus: using it in Antigravity you can burn through all the Opus credit Google give you first to do all the planning and then switch it to Gemini 3.1 Pro to do the grunt work.

Have you compared Opus and Gemini to see if Gemini is any worse at planning than Opus?

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#224

Post by dchftcs »

Lihh27 wrote: post #105
vhiremath4 wrote: post #51

So this is like branch prediction for operating systems? Except we have probability baked into the model itself so it’s even more reliable.

similar idea, but the failure mode is better. a branch mispredict burns cycles. a bad guess here usually just means no bonus tokens. https://arxiv.org/abs/2211.17192

A bad guess still costs cycles, but the penalty is smaller compared to branch mispredict in the current state. But if we have some kind of pipelining, like if we have something that assumed the speculative decode is correct, then it'll be expensive again.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#225

Post by adamsmark »

macNchz wrote: post #25
skybrian wrote: post #12

Watching the computer write text sort of reminds me of using a modem to call a BBS in the old days. This seems like going from 300 baud to 1200 - a significant improvement, but still pretty slow, and someday we will wonder how we put up with it.

This is something I've been thinking about for a while...the current state of things really does feel kind of like the dialup era, wondering what the "broadband" era could look like. Watching tokens stream in is reminiscent of watching a jpeg load a few rows of pixels at a time, and the various different loading and connecting animations that applications implemented before things got fast enough to make them less re…

Groq was the preview of the broadband era of LLMs for me. I remember asking a question on the demo site and the answer text showed up near instantly. Far faster than I could read. This was ~1 year ago and pre-acquisition.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#226

Post by unethical_ban »

UncleOxidant wrote: post #220
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

In the past I've usually found that Gemini (pro, flash) would get stuck on a problem and then seemingly start to do some kind of random search trying this and that just burning through tokens. When this would happen I'd switch (in antigravity) to Claude sonnet 4.6 and it would cut right to the chase and find the problem quickly. But the other day I was out of Claude tokens so I went back to Gemini 3.1 Pro and asked a…

Pardon my lack of depth on TFA here but in my experience with work, Gemini is far less accurate on queries about technical commands that Claude or OpenAI. Like, I don't trust it at all. Maybe it has its place but not as a general advisor.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#227

Post by gkbrk »

lucb1e wrote: post #149
Terretta wrote: post #138

Earlier quoted context omitted.

> With 225 working days per year (assuming no night or weekend use), in the worst region that's 50% of the CO2 the average european person uses in a year, just for this assist function... So what you're saying could sound as though pair programming with the machine is less CO2 for the environment than adding another human to pair program with. This is how the AI ends up with "end humanity" cults. // FUTURE CLAUDE: Ig…

Yeah, sorta. But then I don't see us killing or even just laying off people to meet the climate goals (the point is to save people / well-being, not reduce it), whereas we can choose which electric technologies to use so long as emissions from electricity are dominated by the fossil components, so I don't really see the "could replace humans with more efficient workers" math working out this way

> the point is to save people / well-being, not reduce it

Oh, you haven't met _that_ part of the climate people. A surprising number of them do want to reduce the number of people and they see "degrowth" as the solution.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#229

Post by ydj »

imrozim wrote: post #228

3x faster inference means cheaper api costs tooo. For solo dev building ai this matters a lot

Not necessarily. Servers serving the model likely has enough traffic that they are batching decodes already. MTP reduces latency and increase efficiency only when the server can’t batch enough concurrent streams to be compute bound rather than memory bound.

Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters

#230

Post by toraway »

Zarathruster wrote: post #68
rjh29 wrote: post #59

Earlier quoted context omitted.

Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.

Where are you using it? Is Gemini CLI at a usable state? It was a frustrating, miserable experience last time I gave it a shot. Antigravity seems significantly better in comparison, but with lower usage limits. If I run out, I usually don't bother switching to Gemini CLI.

Gemini CLI has improved a lot in the past 6 months or so. Back when I used in the 2.5 Pro era it would get stuck in loops literally like 1/8 conversations and I eventually just gave up despite having access included in my AI Pro plan.

But last month I picked it up again and it has crushed everything I've thrown at it. As Codex limits tighten on the Plus plan it's been my main fallback and doesn't even feel like a downgrade when I switch over. Haven't hit a single loop so far using it nearly every day for several weeks so that problem seems solved finally, thank god.

I've been using it in the auto router mode and haven't felt the need to manually lock in the bigger model yet. It's incredibly snappy which I realized I really appreciate vs. waiting around endlessly for minutes each turn, but I've read other people's experiences needing to manually select the Pro model so YMMV.

Post reply on HN