Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

blog.google

Post reply on HN
Print view View original

61–70 of 138 posts

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#61

Post by satvikpendem »

redox99 wrote: post #38
ilaksh wrote: post #36

Earlier quoted context omitted.

I think that's probably true for the vast majority of Android phones. But if you have a SOTA expensive beast, I wonder if Gemma 4 12B at 4 bit could work? Maybe something like a Redmagic 11 pro or OnePlus 13 running NanoClaw? But also maybe a few Qwen 3.6 or Qwen 3.5 variants can fit and can handle some simple tasks.

I think Gemma 4 12B is definitely possible to run on high end phones, google claims you need 16GB of memory. But it's probably not very usable, you'll need to swap most stuff other than the LLM. When I tried E2B and E4B with Google Edge Gallery, and added a web search skill from the skill list, E2B would fail (get stuck in a loop), E4B would need a very specific instruction, "weather in [city name]" would not call th…

I'm running a service in production using Gemma 4 models, to get structured JSON output back from web search tool calls using Unsloth Studio and its API, but it did require a rather large and detailed system prompt and tool call healing if the format wasn't JSON for example (retries, reprompting with feeding the error back into the model, etc, this is also what Unsloth Studio does for its self-healing tool call feature). But once I did that, it's been working quite well and on benchmarks I've made, it's about 97% accurate after the first time and basically 100% accurate after retries.

This is running on a server though, not sure how well it'd work on a phone, I should try that. I used AI Edge Gallery on Android and it doesn't seem too good at the web search tool but maybe the web search tool itself, being a community made tool, is pretty bad, because tool calling via Unsloth Studio seems to work just fine with the exact same Gemma models on desktop/server vs the phone.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#62

Post by satvikpendem »

steno132 wrote: post #56
satvikpendem wrote: post #55

Earlier quoted context omitted.

By that logic, any software you run that isn't fully built by yourself is "third party" therefore you shouldn't run anything at all on your machine, thus obviating the need for it entirely.

But practically AI inference requires substantial local computing resources. It's not some web app, it's a order of magnitude more compute needed

Not really, I run a production service on a basic server using these Gemma models, the server is weaker than my MacBook. Most people's laptops and even phones actually can run local models, most simply don't know how. Run Unsloth Studio and you'll see how easy it is.

As the sibling says this is why people want smaller but still performant models.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#63

Post by redox99 »

satvikpendem wrote: post #61
redox99 wrote: post #38

Earlier quoted context omitted.

I think Gemma 4 12B is definitely possible to run on high end phones, google claims you need 16GB of memory. But it's probably not very usable, you'll need to swap most stuff other than the LLM. When I tried E2B and E4B with Google Edge Gallery, and added a web search skill from the skill list, E2B would fail (get stuck in a loop), E4B would need a very specific instruction, "weather in [city name]" would not call th…

I'm running a service in production using Gemma 4 models, to get structured JSON output back from web search tool calls using Unsloth Studio and its API, but it did require a rather large and detailed system prompt and tool call healing if the format wasn't JSON for example (retries, reprompting with feeding the error back into the model, etc, this is also what Unsloth Studio does for its self-healing tool call featu…

I agree that the web search tool probably is pretty bad. However a smart model would never hallucinate impossible weather data if the search tool failed.

I'm sure you can get some out of it if you babysit it with an optimized prompt, harness, etc and you can tolerate some failures. But when I try to run the ChatGPT prompts from my history, even if I pick the easier ones, it's hopeless.

I'd like to have a local agent on the phone with wikipedia level knowledge. But you probably need more like 30B params.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#64

Post by simonw »

reactordev wrote: post #46
simonw wrote: post #21

I just ran one of these locally on a Mac like this: uvx litert-lm run \ --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \ gemma-4-E2B-it.litertlm \ --backend=gpu \ --prompt="Generate an SVG of a pelican riding a bicycle" The first time you run that it downloads 3.2GB to ~/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm It can handle audio and image input too, which is prett…

Not to mention the text-only 0.8GB version. Just crazy. You can have basic real-time conversations on-device that's video and audio aware now.

Have you seen a 0.8GB model file floating around yet? I couldn't find one earlier.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#65

Post by jhatax »

It’s the Friday before WWDC during which Apple is going to announce an “improved” Siri based on Google models (a locked partnership, for now). Maybe it’s a coincidence, but this might be Google releasing models that will be showcased next week by Apple?

No knowledge, just speculation.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#66

Post by satvikpendem »

redox99 wrote: post #63
satvikpendem wrote: post #61

Earlier quoted context omitted.

I'm running a service in production using Gemma 4 models, to get structured JSON output back from web search tool calls using Unsloth Studio and its API, but it did require a rather large and detailed system prompt and tool call healing if the format wasn't JSON for example (retries, reprompting with feeding the error back into the model, etc, this is also what Unsloth Studio does for its self-healing tool call featu…

I agree that the web search tool probably is pretty bad. However a smart model would never hallucinate impossible weather data if the search tool failed. I'm sure you can get some out of it if you babysit it with an optimized prompt, harness, etc and you can tolerate some failures. But when I try to run the ChatGPT prompts from my history, even if I pick the easier ones, it's hopeless. I'd like to have a local agent…

I use the 4B on my phone and it seems to work fine without tool calls. So it's definitely an issue with that and not the model itself. I'll play around and see if I can fix that, you might also try using the Searxng MCP as it's a better web search engine one.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#67

Post by redox99 »

satvikpendem wrote: post #66
redox99 wrote: post #63

Earlier quoted context omitted.

I agree that the web search tool probably is pretty bad. However a smart model would never hallucinate impossible weather data if the search tool failed. I'm sure you can get some out of it if you babysit it with an optimized prompt, harness, etc and you can tolerate some failures. But when I try to run the ChatGPT prompts from my history, even if I pick the easier ones, it's hopeless. I'd like to have a local agent…

I use the 4B on my phone and it seems to work fine without tool calls. So it's definitely an issue with that and not the model itself. I'll play around and see if I can fix that, you might also try using the Searxng MCP as it's a better web search engine one.

I tried most prompts that didn't rely on recent knowledge on the basic "AI Chat", not the "Agent skills" version.

I just tested "List the 5 most recent Argentina vice presidents" on E4B and it literally got all 5 wrong

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#68

Post by thangalin »

dofm wrote: post #27
dist-epoch wrote: post #25

Earlier quoted context omitted.

Google already did https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-un...

This is safetensors. Is there any way to run these on a Mac paired with the MLX QAT? (Pardon my ignorance; this stuff moves so fast)

Did you see this?

https://point.free/blog/gemma-4-on-a-2016-xeon/

Xeon, but could be useful for MTP on Mac.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#69

Post by dofm »

thangalin wrote: post #68
dofm wrote: post #27

Earlier quoted context omitted.

This is safetensors. Is there any way to run these on a Mac paired with the MLX QAT? (Pardon my ignorance; this stuff moves so fast)

Did you see this? https://point.free/blog/gemma-4-on-a-2016-xeon/ Xeon, but could be useful for MTP on Mac.

I hadn't seen this, thanks.

I do have the Qwen 3.6 (35B) MTP implementation running (in LM Studio; it doesn't need a separate drafter), along with non-MTP Gemma 4 26B, and I can see that Unsloth Studio can run the new QAT, but I can't see how you can run the assistant/drafter. Yet.

It's just a constantly changing landscape. Don't get me wrong, it's fascinating and for various reasons I am pleased I can keep up even slightly, but eeeehhh :-)

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#70

Post by satvikpendem »

redox99 wrote: post #67
satvikpendem wrote: post #66

Earlier quoted context omitted.

I use the 4B on my phone and it seems to work fine without tool calls. So it's definitely an issue with that and not the model itself. I'll play around and see if I can fix that, you might also try using the Searxng MCP as it's a better web search engine one.

I tried most prompts that didn't rely on recent knowledge on the basic "AI Chat", not the "Agent skills" version. I just tested "List the 5 most recent Argentina vice presidents" on E4B and it literally got all 5 wrong

I use it for recommendations rather than knowledge, like recipes or basic stuff like that rather than knowledge, I mean it's likely due to its knowledge cutoff so it's not necessarily accurate. But the agent skills section does have a query Wikipedia tool call.

Try this on Unsloth Studio, they seem to have fixed Gemma tool calling.

Post reply on HN