Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

developers.googleblog.com

Post reply on HN
Print view View original

161–170 of 286 posts

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#161

Post by NorwegianDude »

porphyra wrote: post #114

It is funny that Microsoft had been peddling "AI PCs" and Apple had been peddling "made for Apple Intelligence" for a while now, when in fact usable models for consumer GPUs are only barely starting to be a thing on extremely high end GPUs like the 3090.

A 3090 is not a extremely high end GPU. Is a consumer GPU launched in 2020, and even in price and compute it's around a mid-range consumer GPU these days.

The high end consumer card from Nvidia is the RTX 5090, and the professional version of the card is the RTX PRO 6000.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#162

Post by simonw »

prvc wrote: post #148
simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

> ~15GB (MLX) leaving plenty of memory for running other apps. Is that small enough to run well (without thrashing) on a system with only 16GiB RAM?

I expect not. On my Mac at least I've found I need a bunch of GB free to have anything else running at all.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#163

Post by rafaelmn »

999900000999 wrote: post #94

Assuming this can match Claude's latest, and full time usage ( as in you have a system that's constantly running code without any user input,) you'd probably save 600 to 700 a month. A 4090 is only 2K and you'll see an ROI within 90 days. I can imagine this will serve to drive prices for hosted llms lower. At this level any company that produces even a nominal amount of code should be running LMS on prem( AWS if your…

I'd say using a Mac studio with M4 Max and 128 GB RAM will get you way further than 4090 in context size and model size. Cheaper than 2x4090 and less power while being a great overall machine.

I think these consumer GPUs are way too expensive for the amount of memory they pack - and that's intentional price discrimination. Also the builds are gimmicky. It's just not setup for AI models, and the versions that are cost 20k.

AMD has that 128GB RAM strix halo chip but even with soldered ram the bandwidth there is very limited, half of M4 Max, which is half of 4090.

I think this generation of hardware and local models is not there yet - would wait for M5/M6 release.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#164

Post by nico »

Gracana wrote: post #144
nico wrote: post #92

Earlier quoted context omitted.

Thank you for the insights and useful links Will keep experimenting, will also try mistral3.1 edit: just tried mistral3.1 and the quality of the output is very good, at least compared to the other models I tried (llama2:7b-chat, llama2:latest, gemma3:12b, qwq and deepseek-r1:14b) Doing some research, because of their training sets, it seems like most models are not trained on producing long outputs so even if they te…

You can probably find some long-form tuned models on HF. I've had decent results with QwQ-32B (which I can run on my desktop) and Mistral Large (which I have to run on my server). Generating and refining an outline before writing the whole piece can help, and you can also split the piece up into multiple outputs (working a paragraph or two at a time, for instance). So far I've found it to be a tough process, with mix…

Thank you, will try out your suggestions

Have you used something like a director model to supervise the output? If so, could you comment on the effectiveness of it and potentially any tips?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#165

Post by zapnuk »

NorwegianDude wrote: post #161
porphyra wrote: post #114

It is funny that Microsoft had been peddling "AI PCs" and Apple had been peddling "made for Apple Intelligence" for a while now, when in fact usable models for consumer GPUs are only barely starting to be a thing on extremely high end GPUs like the 3090.

A 3090 is not a extremely high end GPU. Is a consumer GPU launched in 2020, and even in price and compute it's around a mid-range consumer GPU these days. The high end consumer card from Nvidia is the RTX 5090, and the professional version of the card is the RTX PRO 6000.

A 3090 still costs 1800€. Thats not mid-range by a long shot

The 5070 or 5070ti are mid range. They cost 650/900€.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#166

Post by mnoronha »

simonw wrote: post #162
prvc wrote: post #148

Earlier quoted context omitted.

> ~15GB (MLX) leaving plenty of memory for running other apps. Is that small enough to run well (without thrashing) on a system with only 16GiB RAM?

I expect not. On my Mac at least I've found I need a bunch of GB free to have anything else running at all.

Any idea why MLX and ollama use such different amounts of ram?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#167

Post by dragonwriter »

NorwegianDude wrote: post #161
porphyra wrote: post #114

It is funny that Microsoft had been peddling "AI PCs" and Apple had been peddling "made for Apple Intelligence" for a while now, when in fact usable models for consumer GPUs are only barely starting to be a thing on extremely high end GPUs like the 3090.

A 3090 is not a extremely high end GPU. Is a consumer GPU launched in 2020, and even in price and compute it's around a mid-range consumer GPU these days. The high end consumer card from Nvidia is the RTX 5090, and the professional version of the card is the RTX PRO 6000.

For model usability as a binary yes/no, pretty much the only dimension that matters is VRAM, and at 24GB the 3090 is still high end for a consumer NVidia GPUs, yes, the 5090 (and only the 5090) is above it, at 32GB, but 24GB is way ahead of the mid-range.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#168

Post by trees101 »

rs186 wrote: post #48
simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…

Not sure how accurate my stats are. I used ollama with the --verbose flag. Using a 4090 and all default settings, I get 40TPS for Gemma 29B model

`ollama run gemma3:27b --verbose` gives me 42.5 TPS +-0.3TPS

`ollama run gemma3:27b-it-qat --verbose` gives me 41.5 TPS +-0.3TPS

Strange results; the full model gives me slightly more TPS.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#169

Post by punnerud »

Just tested the 27B, and it’s not very good at following instructions and is very limited on more complex code problems.

Mapping from one JSON with a lot of plain text, into a new structure and it fails every time.

Ask it to generate SVG, and it’s very simple and almost too dumb.

Nice that it doesn’t need that huge amount of RAM, and perform ok on smaller languages from my initial tests.

Post reply on HN