Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency • Hacker News Board

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#41

Post by adam_arthur » Fri, Jun 05, 2026, 8:36 PM UTC

steno132 wrote: post #40

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

The entire universe of automation projects that can be run effectively for free relative to SoTA models?

I don't think many realize that most LLM embedded automation, pipelines, products will soon be able to run extremely cheaply on models Frontier models will be used for coding/creation use cases, yes. But for all the pseudo-deterministic, pipeline, analysis style things there will be no practical benefit to running frontier models, only additional cost.

Gemma 4 26B outperforms most 100-200B models that I've tested for reasoning and structured output.

Gemma 4 12B can consistently select where to click on browser images given a minimal prompt, and do so very quickly.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#42

Post by Zambyte » Fri, Jun 05, 2026, 8:36 PM UTC

steno132 wrote: post #40

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

I like using my computer.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#43

Post by user2722 » Fri, Jun 05, 2026, 8:36 PM UTC

steno132 wrote: post #40

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

There is tinfoil.sh as well but honestly running this stuff on an airgapped server allows a better peace of mind about the data being used for something else.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#44

Post by mikeocool » Fri, Jun 05, 2026, 8:42 PM UTC

steno132 wrote: post #40

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

> I've been using Claude and GPT models for years

All 3 years?

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#45

Post by jbarrow » Fri, Jun 05, 2026, 8:42 PM UTC

Very impressed with how much the Gemma ecosystem has advanced just this week.

Gemma 12B, multitoken prediction, and official quants released. Feels like Google is putting real effort into this string of releases, and I'm very excited to see that!

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#46

Post by reactordev » Fri, Jun 05, 2026, 8:44 PM UTC

simonw wrote: post #21

I just ran one of these locally on a Mac like this: uvx litert-lm run \ --from-huggingface-repo=litert-community/gemma-4-E2B-it-litert-lm \ gemma-4-E2B-it.litertlm \ --backend=gpu \ --prompt="Generate an SVG of a pelican riding a bicycle" The first time you run that it downloads 3.2GB to ~/.cache/huggingface/hub/models--litert-community--gemma-4-E2B-it-litert-lm It can handle audio and image input too, which is prett…

Not to mention the text-only 0.8GB version. Just crazy. You can have basic real-time conversations on-device that's video and audio aware now.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#47

Post by steno132 » Fri, Jun 05, 2026, 8:45 PM UTC

adam_arthur wrote: post #41

steno132 wrote: post #40

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

The entire universe of automation projects that can be run effectively for free relative to SoTA models? I don't think many realize that most LLM embedded automation, pipelines, products will soon be able to run extremely cheaply on models Frontier models will be used for coding/creation use cases, yes. But for all the pseudo-deterministic, pipeline, analysis style things there will be no practical benefit to running…

Practically if you're running a small personal automation project you're not going to want to waste a lot of time configuring and tuning a local model. You want to build the automation and move on.

If you're building a automation as a company you definitely won't want to take on the long term maintenance overhead of running your own models for some automation project.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#48

Post by steno132 » Fri, Jun 05, 2026, 8:45 PM UTC

mikeocool wrote: post #44

steno132 wrote: post #40

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

> I've been using Claude and GPT models for years All 3 years?

GPT1 was released in 2018, so yes, since then.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#49

Post by adam_arthur » Fri, Jun 05, 2026, 8:45 PM UTC

steno132 wrote: post #47

adam_arthur wrote: post #41

Earlier quoted context omitted.

The entire universe of automation projects that can be run effectively for free relative to SoTA models? I don't think many realize that most LLM embedded automation, pipelines, products will soon be able to run extremely cheaply on models Frontier models will be used for coding/creation use cases, yes. But for all the pseudo-deterministic, pipeline, analysis style things there will be no practical benefit to running…

Practically if you're running a small personal automation project you're not going to want to waste a lot of time configuring and tuning a local model. You want to build the automation and move on. If you're building a automation as a company you definitely won't want to take on the long term maintenance overhead of running your own models for some automation project.

These small models exist in the cloud and are/will be priced commensurately to their size.

Your claim is effectively that companies don't care about operational/cloud costs. Even pre-LLM, companies regularly assessed and tried to pare down cloud spend.

Re: Gemma 4 QAT models: Optimizing compression for mobile and laptop efficiency

#50

Post by steno132 » Fri, Jun 05, 2026, 8:47 PM UTC

user2722 wrote: post #43

steno132 wrote: post #40

I don't get this obsession with smaller models. I've been using Claude and GPT models for years and have had zero issues with them. I see absolutely no benefit to me as a end user for a local model which is going to take up more of my CPU and memory and slow down my machine. I almost always have Internet and if I don't then not having access to a AI model is the least of my concerns.

There is tinfoil.sh as well but honestly running this stuff on an airgapped server allows a better peace of mind about the data being used for something else.

What's wrong with the data being used for something else? Someone is providing digital intelligence to us, saving us many hours a week, so the least we can do is provide them a little data so they are able to improve their service.

It would be selfish and unethical not to in my view. And ultimately the data is just being used in order to improve the models and benefit us, not for anything nefarious.