Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

blog.google

Post reply on HN
Print view View original

21–30 of 138 posts

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#21

Post by simonw »

I just ran one of these locally on a Mac like this:

  uvx litert-lm run \
    --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
    --backend=gpu \
    --prompt="Generate an SVG of a pelican riding a bicycle"
The first time you run that it downloads 3.2GB to ~/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm

It can handle audio and image input too, which is pretty cool for a 3.2GB model. For images:

  uvx litert-lm run \
    --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
    --backend=gpu --vision-backend gpu \
    --attachment image.jpg --prompt describe
And for audio:

  uvx litert-lm run \
    --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \
  gemma-4-E2B-it.litertlm \
    --backend=gpu --audio-backend cpu \
    --attachment audio.wav --prompt transcribe
(The pelican is rubbish, but it's only a 3.2GB file so the fact it even outputs valid SVG is impressive to me: https://gist.github.com/simonw/94b318afde4b1ce5ff67d4b5d0362... )

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#24

Post by redox99 »

I was just testing Gemma E2B and E4B yesterday, and they are just too dumb to be useful outside of niche use cases.

Besides, there's no good agent on Android. Having a model that can't run web searches and browse websites is limited in use, particularly small models that really need to be grounded on search results to be factual, because they can't memorize enough.

Edit: I'd like to know what kind of usage the people that seem to disagree and downvoted this are having.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#27

Post by dofm »

dist-epoch wrote: post #25
WhiteDawn wrote: post #23

Once someone generates a MTP layer for 26B A4B 4 QAT I'll be singing from the hills with my 5 year old GPU.

Google already did https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-un...

This is safetensors. Is there any way to run these on a Mac paired with the MLX QAT?

(Pardon my ignorance; this stuff moves so fast)

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#28

Post by satvikpendem »

llmoorator wrote: post #13
satvikpendem wrote: post #8

Unsloth's collection as well [0], with their results [1]. Looks like they can get very close to 100% accuracy compared to the BF16 model that is unquantized, and Unsloth's quants are better than the original Google's QAT as posted in the article. Personal I'm using the 2B model for web search and structured JSON output back via Unsloth Studio and its API, works very well for that even with the model embedded on phone…

you misunderstand what that chart shows - it shows BF16 QAT Q4_0, not BF16 regular. meaning Google quantized the model to 4 bit and stored the result in BF16 format for compatibility and convenience to downstream packers. Like storing small 8 bit numbers in full 32 bit integers. So it's not close to 100% of unquantized BF16. I'm curious if anybody can explain why Google released 4 bit QAT Q4_0 is not exactly 100% of…

Ah I see, thanks for the clarification.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#30

Post by [deleted] »

Aurornis wrote: post #9
minimaxir wrote: post #2

It's a bit awkward to release Gemma 4 12B ( https://news.ycombinator.com/item?id=48385906 ), and then a canonical Q4_0 Gemma 4 12B a couple days later. It's good that this post lists the expected VRAM usage for the models with Q4_0 Gemma 4 12B being 6.7GB, which will indeed fit Google's claims of fitting within 16GB comfortably, altough it confirms that only the quantized version will do so. Relatedly, in Google's ne…

I'm not sure why you think it's awkward to have multiple releases. It's better to release models and variations as they're ready, not withhold them all until everything is ready to release all at once. The Q4_0 is a quantization aware training checkpoint. It's not a simple quantization of the original Gemma 4 12B.

[deleted]
Post reply on HN