Gemma 4 is really a beast. The 31B version is totally usable like for cases when I'm bored without internet
Accelerating Gemma 4: faster inference with multi-token prediction drafters
71–80 of 345 posts
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#72Sounds like a game changer if I see that kind of speed up on my hardware. So far I've prefered Qwen 3.6 because of its better tool handling, even though Gemma 4 is faster, but I saw they've updated the model template and that's supposed to be better now. Looking forward to trying this with llama.cpp.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#73I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…
Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.
Are you using their TUI, or just their APis in another harness?
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#74I don't see it talked about much, but Gemma (and gemini) use enormously less tokens per output than other models, while still staying within arms reach of top benchmark performance. It's not uncommon to see a gemma vs qwen comparison, where qwen does a bit better, but spent 22 minutes on the task, while gemma aligned the buttons wrong, but only spent 4 minutes on the same prompt. So taken at face value, gemma is now…
Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.
I got really burned by that quality reduction. I subscribed to the AI pro level, and was using it quite a bit, but I stopped because I had to be super attentive to the output because it would make simple mistakes. It was really a shame, because for a while they're Gemini was the best and the AI pro level would allow you enough usage to use it throughout the day as long as you weren't hammering it
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#75So much faster inference with no quality degradation? All that for just some small memory overhead (drafter models are <1B it seems)?
Is it really no quality degradation?
I'm curious where my understanding is wrong, but I didn't think you necessarily got the exact same output with how I understand speculative decoding to be used. I thought that if the small model produces tokens that are "good enough", meaning within the top few tokens the larger model produces, they're accepted.
I thought it doesn't necessarily have to produce the exact same token the larger model would have produced to be accepted (and that requiring this would reduce the hit rate by a lot.) Just one the top model could have produced with whatever top-k and temperature settings.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#76Earlier quoted context omitted.
Groq.
No, it was a custom ASIC chip with weights baked in for a singular model. I do envision a future where we return to cartridges. Local AI is de facto and massively optimised chips are built to be plug and play running a single SoTA model.
Likely https://taalas.com
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#77Earlier quoted context omitted.
Anecdotally the 15/month basic Gemini plan allows coding all day. I'm not hitting the limits or needing to upgrade to 100/month plans like other people are doing with Claude or Codex. Caveat: Gemini has been dumbed down a few times over the last year. Rate limits tightened up too. So it might not be this good in the future.
Where are you using it? Is Gemini CLI at a usable state? It was a frustrating, miserable experience last time I gave it a shot. Antigravity seems significantly better in comparison, but with lower usage limits. If I run out, I usually don't bother switching to Gemini CLI.
As long as you force it to use the pro model and not flash, it is pretty usable. If you go with the default settings though, it will use flash aggressively which results in pretty bad code. I only use it with pro exclusively now.
Even with pro, I have caught it going off the rails a few times. The most frustrating was when I asked it to do translations, and it decided there were too many to do so it wrote a python script that ran locally and used some terrible library to do literal translations, and some of them were downright offensive and sexual in nature. For translations though, Gemini is the best but you have to have it do a sentence or two at a time. If you provide the context around the text, it really knocks it out of the park
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#78these are the updated models: google/gemma-4-31B-it-assistant google/gemma-4-26B-A4B-it-assistant google/gemma-4-E4B-it-assistant google/gemma-4-E2B-it-assistant
for anyone wanting a glossary to explain the naming scheme here:
E4B = 4B effective parameters (using per-layer embeddings)
E2B = 2B (like above)
it = instruction tuned (rlhf and all that jazz)
assistant = Multi-token drafters (the new 2x speed up)
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#79if someone wants to work with gemma and dont deal with ollama or configs - there is (my baby) https://airplane-ai.franzai.com/ Beta but useable
LM Studio (for example) is free, can you pitch me on your USP vs. it?
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#80Watching the computer write text sort of reminds me of using a modem to call a BBS in the old days. This seems like going from 300 baud to 1200 - a significant improvement, but still pretty slow, and someday we will wonder how we put up with it.
There was a startup posted here which built custom hardware that let the AI respond instantly. Thousands of tokens per second.
cerebras
They built an entire wafer ASIC. The entire thing is one huge active ASIC. it takes a lot of cool engineering and cooling to make it work, and is very cool.