Gemma 3 QAT Models: Bringing AI to Consumer GPUs
developers.googleblog.com
Gemma 3 QAT Models: Bringing AI to Consumer GPUs
1–10 of 286 posts
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#2Available on ollama: https://ollama.com/library/gemma3
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#3Could 16gb vram be enough for the 27b QAT version?
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#4Could 16gb vram be enough for the 27b QAT version?
That's what the chart says yes. 14.1GB VRAM usage for the 27B model.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#5First graph is a comparison of the "Elo Score" while using "native" BF16 precision in various models, second graph is comparing VRAM usage between native BF16 precision and their QAT models, but since this method is about doing quantization while also maintaining quality, isn't the obvious graph of comparing the quality between BF16 and QAT missing? The text doesn't seem to talk about it either, yet it's basically the topic of the blog post.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#6Very excited to see these kinds of techniques, I think getting a 30B level reasoning model usable on consumer hardware is going to be a game changer, especially if it uses less power.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#7Available on ollama: https://ollama.com/library/gemma3
Make sure you're using the "-it-qat" suffixed models like "gemma3:27b-it-qat"
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#8Available on ollama: https://ollama.com/library/gemma3
How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern).
The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they knew the right tools.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#9They keep mentioning the RTX 3090 (with 24 GB VRAM), but the model is only 14.1 GB.
Shouldn’t it fit a 5060 Ti 16GB, for instance?
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#10?
Am I missing something?
These have been out for a while; if you follow the HF link you can see, for example, the 27b quant has been downloaded from HF 64,000 times over the last 10 days.
Is there something more to this, or is just a follow up blog post?
(is it just that ollama finally has partial (no images right?) support? Or something else?)