Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

developers.googleblog.com

Post reply on HN
Print view View original

181–190 of 286 posts

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#181

Post by ipdashc »

simonw wrote: post #80
freeamz wrote: post #73

Earlier quoted context omitted.

>I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources. Think it is NOT just you. Most company with decent management also would not want their data going to anything outside the physical server they have in control of. But yeah for most people just use an app and hosted server. But this is HN,there are ppl he…

"Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.

Yeah, this has been confusing me a bit. I'm not complaining by ANY means, but why does it suddenly feel like everyone cares about data privacy in LLM contexts, way more than previous attitudes to allowing data to sit on a bunch of random SaaS products?

I assume because of the assumption that the AI companies will train off of your data, causing it to leak? But I thought all these services had enterprise tiers where they'll promise not to do that?

Again, I'm not complaining, it's good to see people caring about where their data goes. Just interesting that they care now, but not before. (In some ways LLMs should be one of the safer services, since they don't even really need to store any data, they can delete it after the query or conversation is over.)

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#182

Post by sentimentscan »

zapnuk wrote: post #165
NorwegianDude wrote: post #161

Earlier quoted context omitted.

A 3090 is not a extremely high end GPU. Is a consumer GPU launched in 2020, and even in price and compute it's around a mid-range consumer GPU these days. The high end consumer card from Nvidia is the RTX 5090, and the professional version of the card is the RTX PRO 6000.

A 3090 still costs 1800€. Thats not mid-range by a long shot The 5070 or 5070ti are mid range. They cost 650/900€.

A year ago, I bought a brand-new EVGA hybrid-cooled 3090 Ti for 700 euros. I'm still astonished at how good of a decision it was, especially considering the scarcity of 24GB cards available for a similar price. For pure gaming, many cards perform better, but they mostly come with 12 to 16GB of VRAM.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#183

Post by HachiWari8 »

perching_aix wrote: post #70

This is my first time trying to locally host a model - gave both the 12B and 27B QAT models a shot. I was both impressed and disappointed. Setup was piss easy, and the models are great conversationalists. I have a 12 gig card available and the 12B model ran very nice and swift. However, they're seemingly terrible at actually assisting with stuff. Tried something very basic: asked for a powershell one liner to get the…

I tried the 27B QAT model and it hallucinates like crazy. When I ask it for information about some made up person, restaurant, place name, etc., it never says "I don't know about that" and instead seems eager to just make up details. The larger local models like the older Llama 3.3 70B seem better at this, but are also too big to fit on a 24GB GPU.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#184

Post by idonotknowwhy »

parched99 wrote: post #62
holografix wrote: post #3

Could 16gb vram be enough for the 27b QAT version?

I am only able to get the Gemma-3-27b-it-qat-Q4_0.gguf (15.6GB) to run with a 100 token context size on a 5070 ti (16GB) using llamacpp. Prompt Tokens: 10 Time: 229.089 ms Speed: 43.7 t/s Generation Tokens: 41 Time: 959.412 ms Speed: 42.7 t/s

I didn't realise the 5070 is slower than the 3090. Thanks.

If you want a bit more context, try -ctv q8 -ctk q8 (from memory so look it up) to quant the kv cache.

Also an imatrix gguf like iq4xs might be smaller with better quality

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#185

Post by [deleted] »

simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

[deleted]

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#186

Post by danans »

Elucalidavah wrote: post #108
simonw wrote: post #64

Earlier quoted context omitted.

MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.

> MacBook Pro M2 with 64GB of RAM Are there non-mac options with similar capabilities?

Nvidia Orin AGX if a desktop form factor works for you.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#187

Post by orangecat »

trees101 wrote: post #168
rs186 wrote: post #48

Earlier quoted context omitted.

Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…

Not sure how accurate my stats are. I used ollama with the --verbose flag. Using a 4090 and all default settings, I get 40TPS for Gemma 29B model `ollama run gemma3:27b --verbose` gives me 42.5 TPS +-0.3TPS `ollama run gemma3:27b-it-qat --verbose` gives me 41.5 TPS +-0.3TPS Strange results; the full model gives me slightly more TPS.

ollama's `gemma3:27b` is also 4-bit quantized, you need `27b-it-q8_0` for 8 bit or `27b-it-fp16` for FP16. See https://ollama.com/library/gemma3/tags.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#188

Post by zorgmonkey »

paprots wrote: post #133
simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

The original gemma3:27b also took only 22GB using Ollama on my 64GB MacBook. I'm quite confused that the QAT took the same. Do you know why? Which model is better? `gemma3:27b`, or `gemma3:27b-qat`?

Both versions are quantized and should use the same amount of RAM, the difference with QAT is the quantization happens during training time and it should result in slightly better (closer to the bf16 weights) output

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#189

Post by tootie »

nico wrote: post #59
simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

Been super impressed with local models on mac. Love that the gemma models have 128k token context input size. However, outputs are usually pretty short Any tips on generating long output? Like multiple pages of a document, a story, a play or even a book?

I'm using 12b and getting seriously verbose answers. It's squeezed into 8GB and takes its sweet time but answers are really solid.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#190

Post by tootie »

rafaelmn wrote: post #163
999900000999 wrote: post #94

Assuming this can match Claude's latest, and full time usage ( as in you have a system that's constantly running code without any user input,) you'd probably save 600 to 700 a month. A 4090 is only 2K and you'll see an ROI within 90 days. I can imagine this will serve to drive prices for hosted llms lower. At this level any company that produces even a nominal amount of code should be running LMS on prem( AWS if your…

I'd say using a Mac studio with M4 Max and 128 GB RAM will get you way further than 4090 in context size and model size. Cheaper than 2x4090 and less power while being a great overall machine. I think these consumer GPUs are way too expensive for the amount of memory they pack - and that's intentional price discrimination. Also the builds are gimmicky. It's just not setup for AI models, and the versions that are cost…

There's certainly room to grow but I'm running Gemma 12b on a 4060 (8GB VRAM) which I bought for gaming and it's a tad slow but still gives excellent results. And it certainly seems software is outpacing hardware right now. The target is making a good enough model that can run on a phone.
Post reply on HN