Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

developers.googleblog.com

Post reply on HN
Print view View original

151–160 of 286 posts

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#151

Post by parched99 »

tbocek wrote: post #143
parched99 wrote: post #62

Earlier quoted context omitted.

I am only able to get the Gemma-3-27b-it-qat-Q4_0.gguf (15.6GB) to run with a 100 token context size on a 5070 ti (16GB) using llamacpp. Prompt Tokens: 10 Time: 229.089 ms Speed: 43.7 t/s Generation Tokens: 41 Time: 959.412 ms Speed: 42.7 t/s

This is probably due to this: https://github.com/ggml-org/llama.cpp/issues/12637 . This GitHub issue is about interleaved sliding window attention (iSWA) not available in llama.cpp for Gemma 3. This could reduce the memory requirements a lot. They mentioned for a certain scenario, going from 62GB to 10GB.

Resolving that issue, would help reduce (not eliminate) the size of the context. The model will still only just barely fit in 16 GB, which is what the parent comment asked.

Best to have two or more low-end, 16GB GPUs for a total of 32GB VRAM to run most of the better local models.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#153

Post by fer »

ein0p wrote: post #139
lodovic wrote: post #135

Earlier quoted context omitted.

This is a really cool idea. Do you pretrain the model so it can tag people? I have so many photo's that it seems impossible to ever categorize them,using a workflow like yours might help a lot

No, tagging of people is already handled by another model. Gemma just describes what's in the image, and produces a comma separated list of keywords. No additional training is required besides a few tweaks to the prompt so that it outputs just the description, without any "fluff". E.g. it normally prepends such outputs with "Here's a description of the image:" unless you really insist that it should output only the d…

How do you use the keywords after? I have Immich running which does some analysis, but the querying is a bit of a hit and miss.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#154

Post by fragmede »

rob_c wrote: post #129
simonw wrote: post #128

Earlier quoted context omitted.

Sounds like you're excited to see Gemma 3 get the recognition it deserves on Hacker News then.

No just pointing out the flooding obvious as usual and collecting down votes for it

Speaking for myself, my downvotes are not because of the content of your arguments, but because your tone is consistently condescending and dismissive. Comments like “just pointing out the flooding obvious” come off as smug and combative rather than constructive.

HN works best when people engage in good faith, stay curious, and try to move the conversation forward. That kind of tone — even when technically accurate — discourages others from participating and derails meaningful discussion.

If you’re getting downvotes regularly, maybe it's worth considering how your comments are landing with others, not just whether they’re “right.”

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#155

Post by rs186 »

overfeed wrote: post #74
rs186 wrote: post #48

Earlier quoted context omitted.

Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…

> Whereas on local LLM, I watch it painstakingly prints preambles that I don't care about, and get what I actually need after 20 seconds. You may need to "right-size" the models you use to match your hardware, model, and TPS expectations, which may involve using a smaller version of the model with faster TPS, upgrading your jardware, or paying for hosted models. Alternatively, if you can use agentic workflows or tool…

I have a 4070 super for gaming, and used it to play with LLM a few times. It is by no means a bad card, but I realize that unless I want to get 4090 or new Macs that I don't have any other use for, I can only use it to run smaller models. However, most smaller models aren't satisfactory and are still slower than hosted LLMs. I haven't found a model that I am happy with for my hardware.

Regarding agentic workflows -- sounds nice but I am too scared to try it out, based on my experience with standard LLMs like GPT or Claude for writing code. Small snippets or filling in missing unit tests, fine, anything more complicated? Has been a disaster for me.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#156

Post by kgwgk »

paprots wrote: post #133
simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

The original gemma3:27b also took only 22GB using Ollama on my 64GB MacBook. I'm quite confused that the QAT took the same. Do you know why? Which model is better? `gemma3:27b`, or `gemma3:27b-qat`?

Look up 27b in https://ollama.com/library/gemma3/tags

You'll find the id a418f5838eaf which also corresponds to 27b-it-q4_K_M

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#157

Post by ein0p »

fer wrote: post #153
ein0p wrote: post #139

Earlier quoted context omitted.

No, tagging of people is already handled by another model. Gemma just describes what's in the image, and produces a comma separated list of keywords. No additional training is required besides a few tweaks to the prompt so that it outputs just the description, without any "fluff". E.g. it normally prepends such outputs with "Here's a description of the image:" unless you really insist that it should output only the d…

How do you use the keywords after? I have Immich running which does some analysis, but the querying is a bit of a hit and miss.

Search is indeed hit and miss. Immich, for instance, currently does absolutely nothing with the EXIF "description" field, so I store textual descriptions on the side as well. I have found Immich's search by image embeddings to be pretty weak at recall, and even weaker at ranking. IIRC Lightroom Classic (which I also use, but haven't found a way to automate this for without writing an extension) does search that field, but ranking is a bit of a dumpster fire, so your best bet is searching uncommon terms or constraining search by metadata (e.g. not just "black kitten" but "black kitten AND 2025"). I expect this to improve significantly over time - it's a fairly obvious thing to add given the available tech.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#158

Post by dwood_dev »

Elucalidavah wrote: post #108
simonw wrote: post #64

Earlier quoted context omitted.

MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.

> MacBook Pro M2 with 64GB of RAM Are there non-mac options with similar capabilities?

Anything with the Radeon 8060S/Ryzen AI Max+ 395. One of the popular MiniPC Chinese brands has them for preorder[0] with shipping starting May 7th. Framework also has them, but shipping Q3.

0: https://www.gmktec.com/products/prepaid-deposit-amd-ryzen™-a...

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#159

Post by superkuh »

paprots wrote: post #133
simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

The original gemma3:27b also took only 22GB using Ollama on my 64GB MacBook. I'm quite confused that the QAT took the same. Do you know why? Which model is better? `gemma3:27b`, or `gemma3:27b-qat`?

Quantization aware training just means having the model deal with quantized values a bit during training so it handles the quantization better when it is quantized after training/etc. It doesn't change the model size itself.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#160

Post by icedrift »

porphyra wrote: post #114

It is funny that Microsoft had been peddling "AI PCs" and Apple had been peddling "made for Apple Intelligence" for a while now, when in fact usable models for consumer GPUs are only barely starting to be a thing on extremely high end GPUs like the 3090.

Capable local models have been usable on Macs for a while now thanks to their unified memory.
Post reply on HN