Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

blog.google

Post reply on HN
Print view View original

101–110 of 138 posts

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#101

Post by satvikpendem »

mannanj wrote: post #71
steno132 wrote: post #54

Earlier quoted context omitted.

My view is, if you're going to use the service - you should give the data. It's like using Gmail and expecting them not to train their AI models on your data - how can you expect that when they're giving you a secure, reliable, highly functional email client completely for free? The digital economy only works if everyone pays their fair share. If you don't want to give your data then you are really harming everyone b…

However, you didn't actually get what I meant down, so you ended up inadvertently Straw Manning me. My disinterest is in sharing my intellectual IP. Most people up to now, have never shared this much of their intellectual IP with a company. Name one product through human history before that got this much data and insight into human thinking and now can use your most intimate conversations, ideas and needs for non-tra…

Intellectual "property" is not real property. While I disagree with the parent on many things as my comments show, IP is not one of them. Information should be free, for anyone and everyone.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#102

Post by satvikpendem »

mannanj wrote: post #73
steno132 wrote: post #54

Earlier quoted context omitted.

My view is, if you're going to use the service - you should give the data. It's like using Gmail and expecting them not to train their AI models on your data - how can you expect that when they're giving you a secure, reliable, highly functional email client completely for free? The digital economy only works if everyone pays their fair share. If you don't want to give your data then you are really harming everyone b…

Apple is a good example of ethical services. They still give you privacy and ownership of your data, you keep your dignity and data. Google is a horrible model for this - it matches the whole thing about unethical, abusive, gaslighting relationships I described.

The same Apple that takes 30% of all transactions? In reality no corporation is in the public interest.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#103

Post by RandyOrion »

From the perspective of a local llm user, I think the qat doesn't solve the major problem of the gemma models.

Gemma family (gen 1 to gen 4) is consistent with extreme range of activations, i.e., 600000, essentially forcing people to use bf16 kv cache and accept a short context window, e.g., 31b, iq4_xs quantization, 100k context window on 32gb memory. Or, people use q8 kv cache, 200k context window, and accept a large performance penalty.

In contrast, for qwen 3.5 family, the largest activation is below 2000, making q8 or even lower-precision kv cache essentially free estates. Together with linear attention, which doesn't require kv cache, full 262k context window can be easily reached.

Qat training with w4a16 target, while improving performance on inference with low-precision weighs, doesn't solve kv cache problem at all.

In the end, a qat is a qat, and there are unseen efforts behind qat checkpoints. Thank you gemma team for releasing qat checkpoints.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#104

Post by RandyOrion »

RandyOrion wrote: post #103

From the perspective of a local llm user, I think the qat doesn't solve the major problem of the gemma models. Gemma family (gen 1 to gen 4) is consistent with extreme range of activations, i.e., 600000, essentially forcing people to use bf16 kv cache and accept a short context window, e.g., 31b, iq4_xs quantization, 100k context window on 32gb memory. Or, people use q8 kv cache, 200k context window, and accept a lar…

More rants about local inference, consider yourself warned.

Together with bf16 related deliberate hardward degrades on consumer-level nvidia gpus, i.e., gtx 10, rtx 20, 30, 40, 50 series, things gets sour really quickly.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#105

Post by madduci »

reactordev wrote: post #46
simonw wrote: post #21

I just ran one of these locally on a Mac like this: uvx litert-lm run \ --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \ gemma-4-E2B-it.litertlm \ --backend=gpu \ --prompt="Generate an SVG of a pelican riding a bicycle" The first time you run that it downloads 3.2GB to ~/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm It can handle audio and image input too, which is prett…

Not to mention the text-only 0.8GB version. Just crazy. You can have basic real-time conversations on-device that's video and audio aware now.

Where is it? On ollama I see only the bigger one

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#106

Post by dofm »

int_19h wrote: post #84
dofm wrote: post #27

Earlier quoted context omitted.

This is safetensors. Is there any way to run these on a Mac paired with the MLX QAT? (Pardon my ignorance; this stuff moves so fast)

https://huggingface.co/lmstudio-community/gemma-4-26B-A4B-it...

Yeah — that is the base QAT model, and there are safetensors weights for the QAT version of the MTP drafter, but there are no MLX/GGUF versions. I think the answer is a combination of:

1) Gemma 4 MTP is too fresh for off-the-shelf software to use anyway

2) "you can convert them yourself" which is fine, obvs

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#107

Post by viccis »

reactordev wrote: post #46
simonw wrote: post #21

I just ran one of these locally on a Mac like this: uvx litert-lm run \ --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \ gemma-4-E2B-it.litertlm \ --backend=gpu \ --prompt="Generate an SVG of a pelican riding a bicycle" The first time you run that it downloads 3.2GB to ~/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm It can handle audio and image input too, which is prett…

Not to mention the text-only 0.8GB version. Just crazy. You can have basic real-time conversations on-device that's video and audio aware now.

I'll be honest with you. My main ask for on device AI is that when I am typing "Going out for a quick j" it corrects to "jog" and not "Jonathan". I don't think it needs that many gigabytes.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#109

Post by taffydavid »

viccis wrote: post #107
reactordev wrote: post #46

Earlier quoted context omitted.

Not to mention the text-only 0.8GB version. Just crazy. You can have basic real-time conversations on-device that's video and audio aware now.

I'll be honest with you. My main ask for on device AI is that when I am typing "Going out for a quick j" it corrects to "jog" and not "Jonathan". I don't think it needs that many gigabytes.

Who doesn't enjoy a quick Jonathan now and then.

But seriously, wouldn't productive text on a 90s cell phone pass this test?

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#110

Post by taffydavid »

Noob q: can advancements like this targeted at local inference have bonus effects for cloud inference? Presumably if you can get great results on cheaper hardware that also equates to less resource usage on cutting edge hardware, and less power draw?

Will advancements like this ultimately reduce the carbon footprint of AI?

Post reply on HN