Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

blog.google

Post reply on HN
Print view View original

71–80 of 138 posts

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#71

Post by mannanj »

steno132 wrote: post #54
mannanj wrote: post #51

Earlier quoted context omitted.

I don't like the gaslighting of paying Anthropic or Open(Closed)AI and it being said its unsustainable for them to take my payment while simultaneously they take my data (edit: which is incredibly valuable) and I cannot opt out of that. The obsession is for leaving hostile and abusive entities, the corporations or the people who fund them that have a horrible track record in regards to ethicality, rights and respect…

My view is, if you're going to use the service - you should give the data. It's like using Gmail and expecting them not to train their AI models on your data - how can you expect that when they're giving you a secure, reliable, highly functional email client completely for free? The digital economy only works if everyone pays their fair share. If you don't want to give your data then you are really harming everyone b…

However, you didn't actually get what I meant down, so you ended up inadvertently Straw Manning me.

My disinterest is in sharing my intellectual IP. Most people up to now, have never shared this much of their intellectual IP with a company. Name one product through human history before that got this much data and insight into human thinking and now can use your most intimate conversations, ideas and needs for non-training purposes?

You can't even opt out of that! At least for the training data you can opt-out.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#72

Post by redox99 »

satvikpendem wrote: post #70
redox99 wrote: post #67

Earlier quoted context omitted.

I tried most prompts that didn't rely on recent knowledge on the basic "AI Chat", not the "Agent skills" version. I just tested "List the 5 most recent Argentina vice presidents" on E4B and it literally got all 5 wrong

I use it for recommendations rather than knowledge, like recipes or basic stuff like that rather than knowledge, I mean it's likely due to its knowledge cutoff so it's not necessarily accurate. But the agent skills section does have a query Wikipedia tool call. Try this on Unsloth Studio, they seem to have fixed Gemma tool calling.

Argentina vice presidents span from 2007 to 2023. Knowledge cutoff cant explain getting all 5 of them wrong.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#73

Post by mannanj »

steno132 wrote: post #54
mannanj wrote: post #51

Earlier quoted context omitted.

I don't like the gaslighting of paying Anthropic or Open(Closed)AI and it being said its unsustainable for them to take my payment while simultaneously they take my data (edit: which is incredibly valuable) and I cannot opt out of that. The obsession is for leaving hostile and abusive entities, the corporations or the people who fund them that have a horrible track record in regards to ethicality, rights and respect…

My view is, if you're going to use the service - you should give the data. It's like using Gmail and expecting them not to train their AI models on your data - how can you expect that when they're giving you a secure, reliable, highly functional email client completely for free? The digital economy only works if everyone pays their fair share. If you don't want to give your data then you are really harming everyone b…

Apple is a good example of ethical services. They still give you privacy and ownership of your data, you keep your dignity and data. Google is a horrible model for this - it matches the whole thing about unethical, abusive, gaslighting relationships I described.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#74

Post by NicuCalcea »

steno132 wrote: post #50
user2722 wrote: post #43

Earlier quoted context omitted.

There is tinfoil.sh as well but honestly running this stuff on an airgapped server allows a better peace of mind about the data being used for something else.

What's wrong with the data being used for something else? Someone is providing digital intelligence to us, saving us many hours a week, so the least we can do is provide them a little data so they are able to improve their service. It would be selfish and unethical not to in my view. And ultimately the data is just being used in order to improve the models and benefit us, not for anything nefarious.

If sharing our data is the least we can do, they shouldn't also ask us for our money. Otherwise, it's more than the least.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#75

Post by pfheatwole »

WhiteDawn wrote: post #23

Once someone generates a MTP layer for 26B A4B 4 QAT I'll be singing from the hills with my 5 year old GPU.

Models:

- Safetensors: https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-un...

- GGUF: https://huggingface.co/unsloth/gemma-4-26B-A4B-it-GGUF/tree/...

Note the README in the Unsloth list of files: llama.cpp is working on a PR to support the gemma4 drafters: https://github.com/ggml-org/llama.cpp/pull/23398. Also note the PR submitter didn't experience much speedup with 26B (seems typical that MoE models don't generally benefit from MTP).

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#76

Post by reactordev »

simonw wrote: post #64
reactordev wrote: post #46

Earlier quoted context omitted.

Not to mention the text-only 0.8GB version. Just crazy. You can have basic real-time conversations on-device that's video and audio aware now.

Have you seen a 0.8GB model file floating around yet? I couldn't find one earlier.

I think this is the one but it’s 0.8GB VRAM not 0.8GB size.

https://huggingface.co/google/gemma-4-E2B-it-qat-mobile-ct

But they could be cooking up a smaller one because the model card lists the Q_4 quants as being bigger than the mobile or text-only so I think we’ll need to wait for the Q_2_Distilled_Mobile_Textformer version. Still, just amazing work.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#77

Post by rcarmo »

simonw wrote: post #21

I just ran one of these locally on a Mac like this: uvx litert-lm run \ --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \ gemma-4-E2B-it.litertlm \ --backend=gpu \ --prompt="Generate an SVG of a pelican riding a bicycle" The first time you run that it downloads 3.2GB to ~/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm It can handle audio and image input too, which is prett…

Is that actually QAT? the MLX Community models have that in their names, but these don't, and the upload dates don't quite line up.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#78

Post by Melatonic »

redox99 wrote: post #72
satvikpendem wrote: post #70

Earlier quoted context omitted.

I use it for recommendations rather than knowledge, like recipes or basic stuff like that rather than knowledge, I mean it's likely due to its knowledge cutoff so it's not necessarily accurate. But the agent skills section does have a query Wikipedia tool call. Try this on Unsloth Studio, they seem to have fixed Gemma tool calling.

Argentina vice presidents span from 2007 to 2023. Knowledge cutoff cant explain getting all 5 of them wrong.

What did it say were the presidents from those years?

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#79

Post by __mharrison__ »

simonw wrote: post #21

I just ran one of these locally on a Mac like this: uvx litert-lm run \ --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \ gemma-4-E2B-it.litertlm \ --backend=gpu \ --prompt="Generate an SVG of a pelican riding a bicycle" The first time you run that it downloads 3.2GB to ~/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm It can handle audio and image input too, which is prett…

As an aside uvx is so pleasant to use... I wish Nvidia supported it as first-class rather than making folks jump through Docker hoops.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#80

Post by Havoc »

dist-epoch wrote: post #26
somewhatrandom9 wrote: post #16

Could these quantized models make MTP (Multi-Token Prediction) significantly faster when used as drafters for larger regular Gemma 4 models?

Google already released specialized drafters for Gemma 4.

The E2B ones? Or what do you mean by specialized drafters?
Post reply on HN