Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

developers.googleblog.com

Post reply on HN
Print view View original

31–40 of 286 posts

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#31

Post by umajho »

I am currently using the Q4_K_M quantized version of gemma-3-27b-it locally. I previously assumed that a 27B model with image input support wouldn't be very high quality, but after actually using it, the generated responses feel better than those from my previously used DeepSeek-R1-Distill-Qwen-32B (Q4_K_M), and its recognition of images is also stronger than I expected. (I thought the model could only roughly understand the concepts in the image, but I didn't expect it to be able to recognize text within the image.)

Since this article publishes the optimized Q4 quantized version, it would be great if it included more comparisons between the new version and my currently used unoptimized Q4 version (such as benchmark scores).

(I deliberately wrote this reply in Chinese and had gemma-3-27b-it Q4_K_M translate it into English.)

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#33

Post by oezi »

Der_Einzige wrote: post #8
emrah wrote: post #2

Available on ollama: https://ollama.com/library/gemma3

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they kne…

Why is sillytavern the only LLM frontend which matters?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#34

Post by ach9l »

Der_Einzige wrote: post #8
emrah wrote: post #2

Available on ollama: https://ollama.com/library/gemma3

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they kne…

instead of ranting, maybe explain how to make a qat q4 work with images in vllm, afaik it is not yet possible

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#36

Post by oezi »

erichocean wrote: post #13
halflings wrote: post #4

Earlier quoted context omitted.

That's what the chart says yes. 14.1GB VRAM usage for the 27B model.

That's the VRAM required just to load the model weights. To actually use a model, you need a context window. Realistically, you'll want a 20GB GPU or larger, depending on how many tokens you need.

I didn't realize that the context would require such so much memory. Is this KV caches? It would seem like a big advantage if this memory requirement could be reduced.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#37

Post by danielbln »

scosman wrote: post #20
gcr wrote: post #16

Earlier quoted context omitted.

what variant? I’d considered DeepSeek far too large for any consumer GPUs

Some people run Deepseek on CPU. 37B active params - it isn't fast but it's passible.

Actual deepseek or some qwen/llama reasoning fine-tune?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#38

Post by simonw »

noodletheworld wrote: post #19
deepsquirrelnet wrote: post #17

Earlier quoted context omitted.

QAT “quantization aware training” means they had it quantized to 4 bits during training rather than after training in full or half precision. It’s supposedly a higher quality, but unfortunately they don’t show any comparisons between QAT and post-training quantization.

I understand that, but the qat models (1) are not new uploads. How is this more significant now than when they were uploaded 2 weeks ago? Are we expecting new models? I don’t understand the timing. This post feels like it’s two weeks late. [1] - https://huggingface.co/collections/google/gemma-3-qat-67ee61...

The official announcement of the QAT models happened on Friday 18th, two days ago. It looks like they uploaded them to HF in advance of that announcement: https://developers.googleblog.com/en/gemma-3-quantized-aware...

The partnership with Ollama and MLX and LM Studio and llama.cpp was revealed in that announcement, which made the models a lot easier for people to use.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#39

Post by oezi »

Der_Einzige wrote: post #8
emrah wrote: post #2

Available on ollama: https://ollama.com/library/gemma3

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they kne…

Somebody in this thread mentioned 20.x tok/s on ollama. What are you seeing in vLLM?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#40

Post by simonw »

Der_Einzige wrote: post #8
emrah wrote: post #2

Available on ollama: https://ollama.com/library/gemma3

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they kne…

Last I looked vLLM didn't work on a Mac.
Post reply on HN