Does this mean there will be new Gemma 4 models released with MTP, or are they already available in existing models + quants?
They have now been released on e.g Hugging Face with model suffixes "-assistant".
Accelerating Gemma 4: faster inference with multi-token prediction drafters
161–170 of 345 posts
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#162I wonder what latency and tok/s this model on Groq or Cerebras would be capable of. I have a couple LLM driven games [1][2] where speed is really important to the experience. Currently the best performance I can get is the gpt-oss models on Groq or Cerebras but they need quite a bit of extra context and tools to correct for mistakes. I'm making a bet I'll be able to get the same performance much cheaper in the next few months.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#163Google is singlehandedly carrying western open source models. Gemma 4 31B is fantastic. However, it is a little painful to try to fit the best possible version into 24GB vram with vision + this drafter soon. My build doesn't support any more GPUs and I believe I would want another 4090 (overpriced) for best performance or otherwise just replace it altogether.
Qwen is still better that Gemma though. Also you can tune it more for different tasks, which means that you can prioritize thinking and accuracy versus inference speed.
Genuine question: how do you tune it?
I thought "fine-tuning" meant training it on additional data to add additional facts / knowledge? I might be mistaking your use of the word "tune", though :)
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#164Earlier quoted context omitted.
It really is. This is because LLMs with a single output/user are strongly bandwidth limited. Although the hardware can generate multiple tokens simultaneously, it is slowed down if the tokens depend on each other, as is the case with regular text generation. The draft model essentially predicts the next token quickly, enabling you to start generating the subsequent token in parallel. If the guess is right, the second…
> If the guess is right This is the crux. What makes the guess "right"? I think the acceptance criteria is not that the token is exactly the token the big model would have produced. It's accepted of the big model verifies that the probability of that token was high enough. How close it is to the same output (or same distribution of outputs) you'd get from running the big model would be dependent on temperature, top-k…
There is more compute available than bandwidth when computing LLMs.
It's like branch prediction - the CPU predicts what branch you'll take and starts executing it. Later you find out exactly what branch you took. If the prediction was correct, the speculative executed code is kept. If the prediction was wrong, it's thrown away, the pipeline is flushed, and the execution resumes from the branch point.
The same with this thing: 3 tokens, A-B-C were "predicted", you start computing ALL them 3 at the same time, hoping that the prediction checks out. And because of the mathematical structure of the transformer, it costs you almost the same to compute 3 tokens at a time or just one - you are limited by bandwidth, not compute. But CRITICALLY, each token depends on all the previous ones, so if you predicted wrongly one of the tokens, you need to discard all tokens predicted after (flush the pipeline). This is why a prediction is required and why you can't always compute 3 tokens simultaneously - the serial dependency between consecutive tokens. If you were to start computing 3 tokens simultaneously without a prediction, for token C you need to assume some exact values for tokens A and B, but those were not computed yet! But if they were speculatively predicted you can start and hope the prediction was correct.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#165CloudFlare offers excellent service for many of the open-weights models. It's fast, cheap and simple to set up. Can highly suggest as an LLM provider. They serve gemma-4-26b-a4b-it.
It doesn't seem that compelling to me. I can get the gpt-oss models cheaper from the openrouter nitro providers like groq and cerebras. The model you mention on Cloudflare infra is the same price through open router or directly.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#166Earlier quoted context omitted.
As long as you force it to use the pro model and not flash, it is pretty usable. If you go with the default settings though, it will use flash aggressively which results in pretty bad code. I only use it with pro exclusively now. Even with pro, I have caught it going off the rails a few times. The most frustrating was when I asked it to do translations, and it decided there were too many to do so it wrote a python sc…
flash is the fast (duh) model though. its not always beneficial to use pro. in practice: 1/ set to flash 3.1 ; 2/ force to pro...sometimes. mainly when the cli fails to predict what model to use. note that it will sometimes fall back to flash 2, which sucks
Flash will absolutely destroy a complex codebase. It's like a drunk junior programmer. Don't trust it with anything more complex than autocomplete.
Pro is expensive, but good. However they've decreased the pitiful stipend they used to include in even the ultra plan to the point were it's barely usable. I pivoted back to ChatGPT Pro after the recent downgrade they gave Ultra users. Googles Ultra plan cost 2.5x as much and delivers about half the usage.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#167I'm not sure I understand how this work
https://huggingface.co/google/gemma-4-E4B-it-assistant has 78.8M parameters while the standard variant https://huggingface.co/google/gemma-4-E4B-it has 8B parameters.
Is gemma-4-E4B-it-assistant a model I can use stand-alone or a model I need to use in combination with gemma-4-E4B-it?
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#168I'm not sure I understand how this work https://huggingface.co/google/gemma-4-E4B-it-assistant has 78.8M parameters while the standard variant https://huggingface.co/google/gemma-4-E4B-it has 8B parameters. Is gemma-4-E4B-it-assistant a model I can use stand-alone or a model I need to use in combination with gemma-4-E4B-it?
You need the regular gemma model as well. You can think of this as a really small distillation of the original. Useless by its own because it often is wrong, but it is fifth more than not. And because verifying a transformer model can be done faster than running it. We can effectively speed up by using this draft model and only doing the compute where it was wrong.
This is a oversimplification, but tldr you need both yes.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#169MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.
There is a newer PR which will probably be merged soon: https://github.com/ggml-org/llama.cpp/pull/22673
Ohhhh geee!!! I just applied the patch to my local git copy.
You need to use the model on the PR that he submitted, the model is particular because it has extra information that allows the MTP to happen.
I have two amd gpus, and qwen3.6 27B qk6 does around 20t/s generation... If I run it only on one I get like 35t/s.
But with this patch I saw 46t/s with qwen3.6 27B q8... this is insane, it's 250% faster than the original speed, there was no gpu I could upgrade to get that kind of boost, amazing!
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#170don't know about this guy, but qwen3.6:27b with the UD 4bit quant and little-coder/pi has been amazing. the first local LLM experience that can do actual meaningful work