Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
1–10 of 138 posts
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#2It's a bit awkward to release Gemma 4 12B (https://news.ycombinator.com/item?id=48385906), and then a canonical Q4_0 Gemma 4 12B a couple days later.
It's good that this post lists the expected VRAM usage for the models with Q4_0 Gemma 4 12B being 6.7GB, which will indeed fit Google's claims of fitting within 16GB comfortably, altough it confirms that only the quantized version will do so.
Relatedly, in Google's newly released Edge Gallery for macOS, Gemma 4 12B is explicitly listed as unsupported due to not enough RAM even on a 16GB machine, but given the expected VRAM usage here the Q4_0 variant definitely should fit and Google should fix that.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#3had a good run with Gemma 4 E2B Unsloth 4Q: https://youtube.com/shorts/XLsAnz5aAAI
The E4B model doesn’t fit on my phone TPU, so it swaps to RAM, the QAT version means more accuracy, good!
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#4It's a bit awkward to release Gemma 4 12B ( https://news.ycombinator.com/item?id=48385906 ), and then a canonical Q4_0 Gemma 4 12B a couple days later. It's good that this post lists the expected VRAM usage for the models with Q4_0 Gemma 4 12B being 6.7GB, which will indeed fit Google's claims of fitting within 16GB comfortably, altough it confirms that only the quantized version will do so. Relatedly, in Google's ne…
not sure if I understand you, but 4Q and QAT 4Q are different
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#5@google.com'ers, there are no GGUFs (blog says there is)
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#6@google.com'ers, there are no GGUFs (blog says there is)
Isn’t this it? https://huggingface.co/google/gemma-4-12B-it-qat-q4_0-gguf
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#7It's a bit awkward to release Gemma 4 12B ( https://news.ycombinator.com/item?id=48385906 ), and then a canonical Q4_0 Gemma 4 12B a couple days later. It's good that this post lists the expected VRAM usage for the models with Q4_0 Gemma 4 12B being 6.7GB, which will indeed fit Google's claims of fitting within 16GB comfortably, altough it confirms that only the quantized version will do so. Relatedly, in Google's ne…
not sure if I understand you, but 4Q and QAT 4Q are different
It's super annoying when you have products that utilize these because there's...4? releases in 3 weeks?
- Gemma 4 2B/4B/27BE3B/31B
- Gemma 4 2B/4B/27BE3B/31B x "assistant" / MTP drafter models (i.e. multitoken prediction)
- Gemma 4 12B (2 days ago? 1?)
- Gemma 4 QAT 2B/4B/12B/27BE3B/31B x "assistant" models (i.e. multitoken prediction)
It probably sounds silly and really whiny in the abstract. It just causes a ton of work / confusion downstream that feels unnecessary.
Extremely glad for the output, not glad to have to chase it.
ex. llama.cpp currently supports the originals but not the MTP predictors but there is a patch for the MTP predictors but not for the small MoE models and I think it supports the 12B but maybe not media for it yet and now we have these too and the blog says there's GGUFs (llama.cpp models) but there isn't in any of the 12? repos I clicked through. and ~every consumer-facing local LLM app is built on llama.cpp or a fork of it.
Also if anyone at Google is taking feedback over to b/ or product, pleaseeee stop the "E"2B "E"4B thing, unless it's actually taking up less RAM on Android during CPU inference. I can't tell if I need to treat the 4B like an 8B (i.e. beyond most consumer hardware without a GPU) or a 4B (i.e. will run on most consumer hardware since 2021)
EDIT: And, yes, the QAT 12B x mmproj does not work with llama.cpp. I'm glad there's people who have the luxury of not having to, well, actually use these and treat me as whining :) I'll need to schedule another 4-8 hours of work for the 4th time, no fun!
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#8Unsloth's collection as well [0], with their results [1]. Looks like they can get very close to 100% accuracy compared to the BF16 model that is unquantized, and Unsloth's quants are better than the original Google's QAT as posted in the article.
Personal I'm using the 2B model for web search and structured JSON output back via Unsloth Studio and its API, works very well for that even with the model embedded on phones.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#9It's a bit awkward to release Gemma 4 12B ( https://news.ycombinator.com/item?id=48385906 ), and then a canonical Q4_0 Gemma 4 12B a couple days later. It's good that this post lists the expected VRAM usage for the models with Q4_0 Gemma 4 12B being 6.7GB, which will indeed fit Google's claims of fitting within 16GB comfortably, altough it confirms that only the quantized version will do so. Relatedly, in Google's ne…
I'm not sure why you think it's awkward to have multiple releases. It's better to release models and variations as they're ready, not withhold them all until everything is ready to release all at once.
The Q4_0 is a quantization aware training checkpoint. It's not a simple quantization of the original Gemma 4 12B.