Gemma 3 QAT Models: Bringing AI to Consumer GPUs

Post by m00dy » Sun, Apr 20, 2025, 1:34 PM UTC

Der_Einzige wrote: post #8

emrah wrote: post #2

Available on ollama: https://ollama.com/library/gemma3

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they kne…

Post by erichocean » Sun, Apr 20, 2025, 1:34 PM UTC

halflings wrote: post #4

holografix wrote: post #3

Could 16gb vram be enough for the 27b QAT version?

That's what the chart says yes. 14.1GB VRAM usage for the 27B model.

Post by janderson215 » Sun, Apr 20, 2025, 1:37 PM UTC

Der_Einzige wrote: post #8

emrah wrote: post #2

Available on ollama: https://ollama.com/library/gemma3

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they kne…

Post by deepsquirrelnet » Sun, Apr 20, 2025, 1:38 PM UTC

noodletheworld wrote: post #10

? Am I missing something? These have been out for a while; if you follow the HF link you can see, for example, the 27b quant has been downloaded from HF 64,000 times over the last 10 days. Is there something more to this, or is just a follow up blog post? (is it just that ollama finally has partial (no images right?) support? Or something else?)

Post by noodletheworld » Sun, Apr 20, 2025, 1:42 PM UTC

deepsquirrelnet wrote: post #17

noodletheworld wrote: post #10

? Am I missing something? These have been out for a while; if you follow the HF link you can see, for example, the 27b quant has been downloaded from HF 64,000 times over the last 10 days. Is there something more to this, or is just a follow up blog post? (is it just that ollama finally has partial (no images right?) support? Or something else?)

QAT “quantization aware training” means they had it quantized to 4 bits during training rather than after training in full or half precision. It’s supposedly a higher quality, but unfortunately they don’t show any comparisons between QAT and post-training quantization.