Accelerating Gemma 4: faster inference with multi-token prediction drafters
1–10 of 345 posts
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#2I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#3Has anyone managed to get this to work in LM Studio? They've got a option in the UI, but it never seems to allow me to enable it.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#4I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?
[deleted]
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#5I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?
I wonder if for a model that small with a permissive license it might not be worth their time to host a commercial grade inference stack?
Might be easier to chuck it over the fence and let other providers handle it as it'll run in almost any commercial grade card?
Also speculating, but I wonder if it might also create a bit of a pricing problem relative to Gemini flashlight depending on serving cost and quality of outputs?
As a comparison, despite being SotA for their size, the smallest qwen models on openrouter (27b and 35b) are not at all worth using, as there are way bigger and better models for less oricemon a per token basis
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#6I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?
[dead]
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#7nice, will run it later agains qwen3.6 27b, the speed was one of the reasons why in was running qwen and not gemma. the difference was big, there is some magic that happpens when you have more then 100tps.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#8Has anyone managed to get this to work in LM Studio? They've got a option in the UI, but it never seems to allow me to enable it.
It's not implemented in mlx[1] yet (or llama.cpp[2]), so it may take a while.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#9MTP support is being addedto llama.cpp, at least for the Qwen models ( https://github.com/ggml-org/llama.cpp/pull/20533) and I'd imagine Gemma 4 will come soon.
The performance uplift on local/self-hosted models in both quality and speed has been amazing in the last few months.
Re: Accelerating Gemma 4: faster inference with multi-token prediction drafters
#10I find it puzzling Google doesn’t actively promote its own cloud for inference of Gemma 4. Open source is great, love it. But shouldn’t Google want me to be able to use and pay for it through Gemini and vertex?
I wonder if for a model that small with a permissive license it might not be worth their time to host a commercial grade inference stack? Might be easier to chuck it over the fence and let other providers handle it as it'll run in almost any commercial grade card? Also speculating, but I wonder if it might also create a bit of a pricing problem relative to Gemini flashlight depending on serving cost and quality of ou…
i dont know what are you talking about, i replaced an older gpt4o with a finetuned qwen. there is a huge amount of "AI, that can be done with those models, or partly by those models." Huge amount of people would not notice the difference. And if you prepare the context correctly, even bigger slice of people would not notice.