I wonder if they're trying to just kill Anthropic and OpenAI with this, ahah. I assume that Google really doesn't care if the coding model becomes fully overtaken by free models.
Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
131–138 of 138 posts
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#132Earlier quoted context omitted.
0.8GB is for text only. It's more like ~1.1GB if you include video/audio encoder
And your point is what? That’s more than 0.8GB text only if you include more than, text-only?
Their point is that OP used the same dot separated phrase to point out that there's a 0.8GB model and an audio/image model on device. Which reads weird.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#133This is really awesome.
I tried my test-task (generate a Python wrapper for a fairly complex C-interface) with gemma4:26b-a4b-it-qat. And for the first time it would just do it, without prodding and without errors. (Of course Claude just did it a heartbeat, too.)
My optimal local setup now is gemmma4-qat and Q8_0 K/V cache quantization with 256k context windows. And that runs fine with 12GB VRAM and another 10GB in RAM.
Previously I tried with gemma4:26b-a4b-it-q4_K_M and qwen3.6:35b-a3b-q4_K_M, and they both would tie themselves into knots (especially qwen3.6 can take forever with incessant "but wait..." thinking loops.) More often than not, they would not finish the task.
It seems true these 4b QAT models are as precise as Q8_0 quantization (which is supposedly indistinguishable from bf16).
I am really excited about the prospect of local LLM inference.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#134Earlier quoted context omitted.
Another straw man. "real" property or not. You agree that we have some right to our own outputs, right? Is that not dignity, to say "I want my outputs protected". Seems like you think that your ideas should be free, as you called it information. How about you back that up with action... please send me all your most intimate, valuable ideas. Oh no, you don't feel comfortable? Then why are you sharing it with companies…
Sure, it's not like ideas are protected anyway even in the current copyright regime. And anyway I share it with companies because they perform some useful service for me, such as chatting with LLMs about said ideas.
ok got it. I think just because law doesn't recognize something as valuable or worthy of protection, doesn't mean it isn't valuable or worthy of protection. I prefer to aim for the reality and conditions I desire, than accept verbatim what the current reality is even if it's hostile to me and my value.
For example, I think ideas are incredibly valuable as we've discussed. I think theft of them is a norm and then disempowered opinions are conditioned upon the people to make stealing their output easier.
I also share my ideas with companies such as chatting with LLMs but I talk about it because I'm unhappy with it and think I should not forget (and neither should you) that ultimately a valuable asset is being handed to them free. No, actually, I'm paying them for that. And I think that shouldn't be forgotten. When a better alternative is available, or I'm ready, I'm out.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#135llama.cpp has been patched to support Gemma 4 MTP, so thanks to:
https://huggingface.co/RachidAR/gemma-4-26B-A4B-it-qat-assistant-q4_0-gguf
./llama-server \
-hf google/gemma-4-26B-A4B-it-qat-q4_0-gguf \
--spec-draft-hf RachidAR/gemma-4-26B-A4B-it-qat-assistant-q4_0-gguf:Q4_0 \
--spec-type draft-mtp \
--spec-draft-n-max 3
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#136Earlier quoted context omitted.
This is safetensors. Is there any way to run these on a Mac paired with the MLX QAT? (Pardon my ignorance; this stuff moves so fast)
Did you see this? https://point.free/blog/gemma-4-on-a-2016-xeon/ Xeon, but could be useful for MTP on Mac.
To briefly follow up, as of yesterday llama.cpp can do Gemma 4's MTP, so I have this working at least initially — details here:
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#137I wish they would release the base (non instruction tuned) models for use with pattern completion.
they did https://huggingface.co/collections/google/gemma-4
Ah, nice, thanks! https://huggingface.co/google/gemma-4-12B/tree/main No .gguf but I guess I can convert the safetensors myself like it's 2023 again.
Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency
#138Earlier quoted context omitted.
Sure, it's not like ideas are protected anyway even in the current copyright regime. And anyway I share it with companies because they perform some useful service for me, such as chatting with LLMs about said ideas.
ok got it. I think just because law doesn't recognize something as valuable or worthy of protection, doesn't mean it isn't valuable or worthy of protection. I prefer to aim for the reality and conditions I desire, than accept verbatim what the current reality is even if it's hostile to me and my value. For example, I think ideas are incredibly valuable as we've discussed. I think theft of them is a norm and then dise…
I desire lack of copyright and IP protections for anyone and everyone including corporations, therefore we fundamentally didagree; as the old saying goes, ideas are worthless, execution is everything.