Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

blog.google

Post reply on HN
Print view View original

11–20 of 138 posts

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#11

Post by ddarolfi »

refulgentis wrote: post #7
netdur wrote: post #4

Earlier quoted context omitted.

not sure if I understand you, but 4Q and QAT 4Q are different

It's super annoying when you have products that utilize these because there's...4? releases in 3 weeks? - Gemma 4 2B/4B/27BE3B/31B - Gemma 4 2B/4B/27BE3B/31B x "assistant" / MTP drafter models (i.e. multitoken prediction) - Gemma 4 12B (2 days ago? 1?) - Gemma 4 QAT 2B/4B/12B/27BE3B/31B x "assistant" models (i.e. multitoken prediction) It probably sounds silly and really whiny in the abstract. It just causes a ton of…

These models aren't products? They are open source ish (open weight I guess), research outputs. While the naming scheme may be confusing, it is relevant and important. I believe it's on you to understand it.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#12

Post by satvikpendem »

refulgentis wrote: post #7
netdur wrote: post #4

Earlier quoted context omitted.

not sure if I understand you, but 4Q and QAT 4Q are different

It's super annoying when you have products that utilize these because there's...4? releases in 3 weeks? - Gemma 4 2B/4B/27BE3B/31B - Gemma 4 2B/4B/27BE3B/31B x "assistant" / MTP drafter models (i.e. multitoken prediction) - Gemma 4 12B (2 days ago? 1?) - Gemma 4 QAT 2B/4B/12B/27BE3B/31B x "assistant" models (i.e. multitoken prediction) It probably sounds silly and really whiny in the abstract. It just causes a ton of…

Just use Unsloth Studio it supports them all.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#13

Post by llmoorator »

satvikpendem wrote: post #8

Unsloth's collection as well [0], with their results [1]. Looks like they can get very close to 100% accuracy compared to the BF16 model that is unquantized, and Unsloth's quants are better than the original Google's QAT as posted in the article. Personal I'm using the 2B model for web search and structured JSON output back via Unsloth Studio and its API, works very well for that even with the model embedded on phone…

you misunderstand what that chart shows - it shows BF16 QAT Q4_0, not BF16 regular.

meaning Google quantized the model to 4 bit and stored the result in BF16 format for compatibility and convenience to downstream packers.

Like storing small 8 bit numbers in full 32 bit integers.

So it's not close to 100% of unquantized BF16.

I'm curious if anybody can explain why Google released 4 bit QAT Q4_0 is not exactly 100% of BF16 QAT Q4_0? seems like it should be just bit twiddling, no further quantization to convert between these two packings. Unsloth talks about "lattice alignment" being an issue.

That being said I hate it that smol model makers, like Google, Qwen, ... only show the BF16 benchmarks when they release a new models, knowing that what people really run are 4-8 bit quantizations, so it's really hard to understand how much you lose when you run 4 bit vs 6 bit...

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#14

Post by slopinthebag »

satvikpendem wrote: post #8

Unsloth's collection as well [0], with their results [1]. Looks like they can get very close to 100% accuracy compared to the BF16 model that is unquantized, and Unsloth's quants are better than the original Google's QAT as posted in the article. Personal I'm using the 2B model for web search and structured JSON output back via Unsloth Studio and its API, works very well for that even with the model embedded on phone…

I'm confused, the unsloth model is ~600mb and the one from google is 7gb?

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#17

Post by refulgentis »

ddarolfi wrote: post #11
refulgentis wrote: post #7

Earlier quoted context omitted.

It's super annoying when you have products that utilize these because there's...4? releases in 3 weeks? - Gemma 4 2B/4B/27BE3B/31B - Gemma 4 2B/4B/27BE3B/31B x "assistant" / MTP drafter models (i.e. multitoken prediction) - Gemma 4 12B (2 days ago? 1?) - Gemma 4 QAT 2B/4B/12B/27BE3B/31B x "assistant" models (i.e. multitoken prediction) It probably sounds silly and really whiny in the abstract. It just causes a ton of…

These models aren't products? They are open source ish (open weight I guess), research outputs. While the naming scheme may be confusing, it is relevant and important. I believe it's on you to understand it.

I understand it. :)

And you're absolutely right to point out they aren't products - I hoped that was clear - when you're building a product with them, you end up having to do the same build loop 4 times, in this instance :)

Post reply on HN