Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

blog.google

Post reply on HN
Print view View original

121–130 of 138 posts

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#121

Post by ComputerGuru »

mft_ wrote: post #113
satvikpendem wrote: post #8

Unsloth's collection as well [0], with their results [1]. Looks like they can get very close to 100% accuracy compared to the BF16 model that is unquantized, and Unsloth's quants are better than the original Google's QAT as posted in the article. Personal I'm using the 2B model for web search and structured JSON output back via Unsloth Studio and its API, works very well for that even with the model embedded on phone…

Is this [0] saying that unsloth's versions of Google's QAT models are better than Google's own QAT models? Or am I not understanding it correctly? [0] https://unsloth.ai/docs/models/gemma-4/qat#qat-analysis

It's saying it's better than naively truncating the QAT release to 4 bits.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#122

Post by itake »

illusive4080 wrote: post #98
jhatax wrote: post #65

It’s the Friday before WWDC during which Apple is going to announce an “improved” Siri based on Google models (a locked partnership, for now). Maybe it’s a coincidence, but this might be Google releasing models that will be showcased next week by Apple? No knowledge, just speculation.

As an amateur app dev using on device AI: If they replace Apple Foundation model with Gemma 4 I would be so happy.

I’m curious like what performance if we met you would expect and why?

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#123

Post by victorbjorklund »

steno132 wrote: post #48
mikeocool wrote: post #44

Earlier quoted context omitted.

> I've been using Claude and GPT models for years All 3 years?

GPT1 was released in 2018, so yes, since then.

GPT1 was way worse than small Gemma’s are now.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#125

Post by reactordev »

yalok wrote: post #99
reactordev wrote: post #46

Earlier quoted context omitted.

Not to mention the text-only 0.8GB version. Just crazy. You can have basic real-time conversations on-device that's video and audio aware now.

0.8GB is for text only. It's more like ~1.1GB if you include video/audio encoder

And your point is what? That’s more than 0.8GB text only if you include more than, text-only?

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#126

Post by reactordev »

madduci wrote: post #105
reactordev wrote: post #46

Earlier quoted context omitted.

Not to mention the text-only 0.8GB version. Just crazy. You can have basic real-time conversations on-device that's video and audio aware now.

Where is it? On ollama I see only the bigger one

I don’t use ollama, can you pull from HF?

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#127

Post by sowbug »

steno132 wrote: post #40

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

Whatever you're doing, try doing 500 or 1,000 of it in a batch. You'll exhaust any subscription quota you have, or if you're paying per token, you will probably find it too expensive. That's when you'll start to ask "how smart a model do I really need for this job?", and you'll investigate running a small but sufficiently capable model on your own PC, churning overnight through your 1,000 tasks.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#128

Post by dofm »

3abiton wrote: post #86
coder543 wrote: post #33

Earlier quoted context omitted.

> meaning Google quantized the model to 4 bit and stored the result in BF16 format for compatibility and convenience to downstream packers. You also misunderstand what is happening. Google did not do that. Google further trained the original model with an objective of minimizing error when quantized to 4-bit. The BF16 QAT is not an upscaled 4-bit model. When quantized to 4-bit, it should lose less accuracy than a typ…

Are there evidence that this approach helps maintain "accuracy" performance when quantized? It sounds a bit like mxfp4 with gpt-oss, which was a confusing model upon release.

I have just been humbled by the Gemma 4 26B QAT build (unsloth's version), which insisted repeatedly that I am wrong in my requirements for some niche wordpress code, which cannot be satisfied.

I am a good WP developer so I kept prodding it and it kept insisting, and it explained with clarity. Turns out it is right and I was wrong, as I would have found out if I'd written the code myself.

I've been using this particular test for days, experimenting in ways to generate and prompt code. The 4-bit quantisation of the pre-QAT model does not catch this error. And nor can the Qwen 3.6 sparse model, which confidently blazed past it and never mentioned it.

(FWIW neither did plain ChatGPT; maybe Codex would)

Anecdotal, but there you go. I am somewhat weirded out by it.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#129

Post by satvikpendem »

mannanj wrote: post #119
satvikpendem wrote: post #101

Earlier quoted context omitted.

Intellectual "property" is not real property. While I disagree with the parent on many things as my comments show, IP is not one of them. Information should be free, for anyone and everyone.

Another straw man. "real" property or not. You agree that we have some right to our own outputs, right? Is that not dignity, to say "I want my outputs protected". Seems like you think that your ideas should be free, as you called it information. How about you back that up with action... please send me all your most intimate, valuable ideas. Oh no, you don't feel comfortable? Then why are you sharing it with companies…

Sure, it's not like ideas are protected anyway even in the current copyright regime. And anyway I share it with companies because they perform some useful service for me, such as chatting with LLMs about said ideas.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#130

Post by robgough »

jhatax wrote: post #65

It’s the Friday before WWDC during which Apple is going to announce an “improved” Siri based on Google models (a locked partnership, for now). Maybe it’s a coincidence, but this might be Google releasing models that will be showcased next week by Apple? No knowledge, just speculation.

I'm fully expecting an updated foundation model of some kind, but I would bet money they don't utter the words "Google", "Gemini", or "Gemma" even once.
Post reply on HN