Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

developers.googleblog.com

Post reply on HN
Print view View original

171–180 of 286 posts

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#171

Post by dragonwriter »

porphyra wrote: post #114

It is funny that Microsoft had been peddling "AI PCs" and Apple had been peddling "made for Apple Intelligence" for a while now, when in fact usable models for consumer GPUs are only barely starting to be a thing on extremely high end GPUs like the 3090.

AI PCs aren't about running the kind of models that take a 3090-class GPU, or even running on GPU at all, but systems where the local end is running something like Phi-3.5-vision-instruct, on system RAM using a CPU with an integrated NPU, which is why the AI PC requirements specify an NPU, a certain amount of processing capacity, and a minimum amount of DDR5/LPDDR5 system RAM.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#172

Post by codybontecou »

simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

Can you run the mlx-variation of this model through Ollama so that I can interact with it in Open WebUI?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#173

Post by NorwegianDude »

zapnuk wrote: post #165
NorwegianDude wrote: post #161

Earlier quoted context omitted.

A 3090 is not a extremely high end GPU. Is a consumer GPU launched in 2020, and even in price and compute it's around a mid-range consumer GPU these days. The high end consumer card from Nvidia is the RTX 5090, and the professional version of the card is the RTX PRO 6000.

A 3090 still costs 1800€. Thats not mid-range by a long shot The 5070 or 5070ti are mid range. They cost 650/900€.

3090s are no longer produced, that's why new ones are so expensive. At least here, used 3090s are around €650, and a RTX 5070 is around €625.

It's definitely not extremely high end any more, the price is(at least here) the same as the new mid range consumer cards.

I guess the price can vary by location, but €1800 for a 3090 is crazy, that's more than the new price in 2020.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#174

Post by simonw »

codybontecou wrote: post #172
simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

Can you run the mlx-variation of this model through Ollama so that I can interact with it in Open WebUI?

I haven't tried it yet but there's an MLX project that exposes an OpenAI-compatible serving endpoint that should work with Open WebUI: https://github.com/madroidmaq/mlx-omni-server

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#175

Post by NorwegianDude »

dragonwriter wrote: post #167
NorwegianDude wrote: post #161

Earlier quoted context omitted.

A 3090 is not a extremely high end GPU. Is a consumer GPU launched in 2020, and even in price and compute it's around a mid-range consumer GPU these days. The high end consumer card from Nvidia is the RTX 5090, and the professional version of the card is the RTX PRO 6000.

For model usability as a binary yes/no, pretty much the only dimension that matters is VRAM, and at 24GB the 3090 is still high end for a consumer NVidia GPUs, yes, the 5090 (and only the 5090) is above it, at 32GB, but 24GB is way ahead of the mid-range.

24 GB of VRAM is a large amount of VRAM on a consumer GPU, that I totally agree with you on. But it's definitely not an extremely high end GPU these days. It is suitable, yes, but not high end. The high end alternative for a consumer GPU would be the RTX 5090, but that is only available for €3000 now, while used 3090s are around €650.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#176

Post by starik36 »

rs186 wrote: post #48
simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…

On an A5000 with 24GB, this model typically gets between 20 to 25 tps.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#177

Post by Der_Einzige »

Zambyte wrote: post #138
Der_Einzige wrote: post #105

Earlier quoted context omitted.

Running any HF model on vllm is as simple as pasting a model name into one command in your terminal.

What command is it? Because that was not at all my experience.

Vllm serve… huggingface gives run instructions for every model with vllm on their website.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#178

Post by starik36 »

ein0p wrote: post #127
rs186 wrote: post #48

Earlier quoted context omitted.

Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…

Sometimes TPS doesn't matter. I've generated textual descriptions for 100K or so images in my photo archive, some of which I have absolutely no interest in uploading to someone else's computer. This works pretty well with Gemma. I use local LLMs all the time for things where privacy is even remotely important. I estimate this constitutes easily a quarter of my LLM usage.

I was thinking of doing the same, but I would like to include people's name. in the description. For example "Jennifer looking out in the desert sky.".

As it stands, Gemma will just say "Woman looking out in the desert sky."

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#179

Post by PhilippGille »

miki123211 wrote: post #91

What would be the best way to deploy this if you're maximizing for GPU utilization in a multi-user (API) scenario? Structured output support would be a big plus. We're working with a GPU-poor organization with very strict data residency requirements, and these models might be exactly what we need. I would normally say VLLM, but the blog post notably does not mention VLLM support.

vLLM lists Gemma 3 as supported, if I'm not mistaken: https://docs.vllm.ai/en/latest/models/supported_models.html#...

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#180

Post by Zambyte »

Der_Einzige wrote: post #177
Zambyte wrote: post #138

Earlier quoted context omitted.

What command is it? Because that was not at all my experience.

Vllm serve… huggingface gives run instructions for every model with vllm on their website.

How do I serve multiple models? I can pick from dozens of models that I have downloaded through Open WebUI.
Post reply on HN