For a moment I got excited thinking QAT is Intel Quick Assist Technology...
Same I had to do a double take. Would be pretty humourous if they somehow took advantage of crypto offloading to accelerate ai inference
Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
31–40 of 138 posts
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#32Unsloth's collection as well [0], with their results [1]. Looks like they can get very close to 100% accuracy compared to the BF16 model that is unquantized, and Unsloth's quants are better than the original Google's QAT as posted in the article. Personal I'm using the 2B model for web search and structured JSON output back via Unsloth Studio and its API, works very well for that even with the model embedded on phone…
I'm confused, the unsloth model is ~600mb and the one from google is 7gb?
One is quantized, the other one is Quantization-ready.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#33Unsloth's collection as well [0], with their results [1]. Looks like they can get very close to 100% accuracy compared to the BF16 model that is unquantized, and Unsloth's quants are better than the original Google's QAT as posted in the article. Personal I'm using the 2B model for web search and structured JSON output back via Unsloth Studio and its API, works very well for that even with the model embedded on phone…
you misunderstand what that chart shows - it shows BF16 QAT Q4_0, not BF16 regular. meaning Google quantized the model to 4 bit and stored the result in BF16 format for compatibility and convenience to downstream packers. Like storing small 8 bit numbers in full 32 bit integers. So it's not close to 100% of unquantized BF16. I'm curious if anybody can explain why Google released 4 bit QAT Q4_0 is not exactly 100% of…
> meaning Google quantized the model to 4 bit and stored the result in BF16 format for compatibility and convenience to downstream packers.
You also misunderstand what is happening. Google did not do that. Google further trained the original model with an objective of minimizing error when quantized to 4-bit. The BF16 QAT is not an upscaled 4-bit model. When quantized to 4-bit, it should lose less accuracy than a typical 16-bit model loses when quantized to 4-bit, but the loss is not zero, because it is not based on a 4-bit model.
The Gemma 3 QAT report was a bit clearer:
https://developers.googleblog.com/en/gemma-3-quantized-aware...
"Instead of just quantizing the model after it's fully trained, QAT incorporates the quantization process during training. QAT simulates low-precision operations during training to allow quantization with less degradation afterwards for smaller, faster models while maintaining accuracy. Diving deeper, we applied QAT on ~5,000 steps using probabilities from the non-quantized checkpoint as targets. We reduce the perplexity drop by 54% (using llama.cpp perplexity evaluation) when quantizing down to Q4_0."
The BF16 is just trained to be more resistant to simulated quantization, which helps when it is actually quantized. Google is not doing post-training on the 4-bit model directly.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#34Earlier quoted context omitted.
These models aren't products? They are open source ish (open weight I guess), research outputs. While the naming scheme may be confusing, it is relevant and important. I believe it's on you to understand it.
I understand it. :) And you're absolutely right to point out they aren't products - I hoped that was clear - when you're building a product with them , you end up having to do the same build loop 4 times, in this instance :)
You can stop after the first one. Choosing to repeat the process is on you, and probably because you see some benefit in using the variant(s) you build on top of.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#35Earlier quoted context omitted.
These models aren't products? They are open source ish (open weight I guess), research outputs. While the naming scheme may be confusing, it is relevant and important. I believe it's on you to understand it.
I understand it. :) And you're absolutely right to point out they aren't products - I hoped that was clear - when you're building a product with them , you end up having to do the same build loop 4 times, in this instance :)
Yes my framing was a little confusing. You were clear in that you are building products on them.
I was more saying that because these gemma models are not products, and instead research outputs, the naming scheme should be more scientific rather than consumer friendly.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#36I was just testing Gemma E2B and E4B yesterday, and they are just too dumb to be useful outside of niche use cases. Besides, there's no good agent on Android. Having a model that can't run web searches and browse websites is limited in use, particularly small models that really need to be grounded on search results to be factual, because they can't memorize enough. Edit: I'd like to know what kind of usage the people…
I think that's probably true for the vast majority of Android phones. But if you have a SOTA expensive beast, I wonder if Gemma 4 12B at 4 bit could work? Maybe something like a Redmagic 11 pro or OnePlus 13 running NanoClaw?
But also maybe a few Qwen 3.6 or Qwen 3.5 variants can fit and can handle some simple tasks.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#37Being able to run the 12B on 8gb VRAM is huge. It's crazy to see how fast these small local models have evolved.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#38I was just testing Gemma E2B and E4B yesterday, and they are just too dumb to be useful outside of niche use cases. Besides, there's no good agent on Android. Having a model that can't run web searches and browse websites is limited in use, particularly small models that really need to be grounded on search results to be factual, because they can't memorize enough. Edit: I'd like to know what kind of usage the people…
I think that's probably true for the vast majority of Android phones. But if you have a SOTA expensive beast, I wonder if Gemma 4 12B at 4 bit could work? Maybe something like a Redmagic 11 pro or OnePlus 13 running NanoClaw? But also maybe a few Qwen 3.6 or Qwen 3.5 variants can fit and can handle some simple tasks.
I think Gemma 4 12B is definitely possible to run on high end phones, google claims you need 16GB of memory. But it's probably not very usable, you'll need to swap most stuff other than the LLM.
When I tried E2B and E4B with Google Edge Gallery, and added a web search skill from the skill list, E2B would fail (get stuck in a loop), E4B would need a very specific instruction, "weather in [city name]" would not call the web search tool, I'd need "web search weather in [city name]". And the result was completely hallucinated and impossible. It claimed 14c and feels like 4c (which is impossible), and 10% humidity (which is almost impossible in this city)
Asking wikipedia level history questions (without any tool use), the results were awful as well.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#39had a good run with Gemma 4 E2B Unsloth 4Q: https://youtube.com/shorts/XLsAnz5aAAI The E4B model doesn’t fit on my phone TPU, so it swaps to RAM, the QAT version means more accuracy, good!
How do you know it swaps to ram vs on the TPU?
Would be interested in testing this on my pixel.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#40I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them.
I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.