Is Google's local model strategy tuned to pegging down big AI cloud labs a notch?
Accelerating Gemma 4: faster inference with multi-token prediction drafters
31–40 of 345 posts
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#32MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533 ) and I'd imagine Gemma 4 will come soon. The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.
Yeah important conceptually to remember MTP is kind of just more weights, but speculative decoding is the runtime algorithm that’s a significant add to whatever code is serving the model.
That is.. inaccurate.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#33ok so? Anyone got a verdict/review?
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#34CloudFlare offers excellent service for many of the open-weights models. It's fast, cheap and simple to set up. Can highly suggest as an LLM provider.
They serve gemma-4-26b-a4b-it.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#35So much faster inference with no quality degradation? All that for just some small memory overhead (drafter models are <1B it seems)?
They also published draft models for E4B and E2B. For those, the draft models are only 78m parameters: https://huggingface.co/google/gemma-4-E4B-it-assistant
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#36So much faster inference with no quality degradation? All that for just some small memory overhead (drafter models are <1B it seems)?
MTP requires a separate KV cache, so there is more memory overhead than just the weights of the MTP model, but it's a manageable amount.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#37CloudFlare offers excellent service for many of the open-weights models. It's fast, cheap and simple to set up. Can highly suggest as an LLM provider. They serve gemma-4-26b-a4b-it.
They do indeed. See https://developers.cloudflare.com/workers-ai/models/ They seem to allow some free usage without user account. Do they list limits anywhere?
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#38I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?
What do you mean? It just works with Google AI Studio.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#39Watching the computer write text sort of reminds me of using a modem to call a BBS in the old days. This seems like going from 300 baud to 1200 - a significant improvement, but still pretty slow, and someday we will wonder how we put up with it.
Check chatjimmy.ai
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#40Watching the computer write text sort of reminds me of using a modem to call a BBS in the old days. This seems like going from 300 baud to 1200 - a significant improvement, but still pretty slow, and someday we will wonder how we put up with it.
You're right about it being reminiscent of the dial-up area, but I don't believe it's 300 to 1200; it's more like 4800:
Modem vs Claude according to Claude:
300 @ 2368 characters - 1m 19s
1200 @ 2368 characters - 19.7s
2400 @ 2368 characters - 9.9s
14.4K @ 2368 characters - 1.6s
33.6K @ 2368 characters - 705 ms
56K @ 2368 characters - 447 ms
Claude @ 2368 characters - 7.9s