Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

developers.googleblog.com

Post reply on HN
Print view View original

201–210 of 286 posts

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#201

Post by bobjordan »

simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

Thanks for the call out on this model! I have 42gb usable VRAM on my ancient (~10yrs old) quad-sli titan-x workstation and have been looking for a model to balance large context window with output quality. I'm able to run this model with a 56K context window and it just fits into my 42gb VRAM to run 100% GPU. The output quality is really good and 56K context window is very usable. Nice find!

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#203

Post by vbezhenar »

pornel wrote: post #192
ipdashc wrote: post #181

Earlier quoted context omitted.

Yeah, this has been confusing me a bit. I'm not complaining by ANY means, but why does it suddenly feel like everyone cares about data privacy in LLM contexts, way more than previous attitudes to allowing data to sit on a bunch of random SaaS products? I assume because of the assumption that the AI companies will train off of your data, causing it to leak? But I thought all these services had enterprise tiers where t…

It is due to the risk of a leak. Laundering of data through training makes it a more complicated case than a simple data theft or copyright infringement. Leaks could be accidental, e.g. due to an employee logging in to their free-as-in-labor personal account instead of a no-training Enterprise account. It's safer to have a complete ban on providers that may collect data for training.

How can you be sure that AWS will not use your data to train their models? They got enormous data, probably most data in the world.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#204

Post by jayavanth »

perching_aix wrote: post #70

This is my first time trying to locally host a model - gave both the 12B and 27B QAT models a shot. I was both impressed and disappointed. Setup was piss easy, and the models are great conversationalists. I have a 12 gig card available and the 12B model ran very nice and swift. However, they're seemingly terrible at actually assisting with stuff. Tried something very basic: asked for a powershell one liner to get the…

you should set a lower temperature

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#205

Post by justanotheratom »

nico wrote: post #61
justanotheratom wrote: post #49

Anyone packaged one of these in an iPhone App? I am sure it is doable, but I am curious what tokens/sec is possible these days. I would love to ship "private" AI Apps if we can get reasonable tokens/sec.

What kind of functionality do you need from the model? For basic conversation and RAG, you can use tinyllama or qwen-2.5-0.5b, both of which run on a raspberry pi at around 5-20 tokens per second

I am looking for structured output at about 100-200 tokens/second on iPhone 14+. Any pointers?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#206

Post by technologesus »

Just for fun I created a new personal benchmark for vision-enabled LLMs: playing minecraft. I used JSON structured output in LM Studio to create basic controls for the game. Unfortunately no matter how hard I proompted, gemma-3-27b QAT is not really able to understand simple minecraft scenarios. It would say things like "I'm now looking at a stone block. I need to break it" when it is looking out at the horizon in the desert.

Here is the JSON schema: https://pastebin.com/SiEJ6LEz System prompt: https://pastebin.com/R68QkfQu

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#208

Post by rob_c »

fragmede wrote: post #154
rob_c wrote: post #129

Earlier quoted context omitted.

No just pointing out the flooding obvious as usual and collecting down votes for it

Speaking for myself, my downvotes are not because of the content of your arguments, but because your tone is consistently condescending and dismissive. Comments like “just pointing out the flooding obvious” come off as smug and combative rather than constructive. HN works best when people engage in good faith, stay curious, and try to move the conversation forward. That kind of tone — even when technically accurate —…

My tone only switches once people get uppity. The original comment is on point and accurate, not combative and not insulting (unless the community seriously takes a 'lol'....

Tbh I give up writing that in response to this rant. My polite poke holds and it's non insulting so I'm not going to capitulate to those childish enough to not look inwards.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#209

Post by mjlee »

simonw wrote: post #80
freeamz wrote: post #73

Earlier quoted context omitted.

>I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources. Think it is NOT just you. Most company with decent management also would not want their data going to anything outside the physical server they have in control of. But yeah for most people just use an app and hosted server. But this is HN,there are ppl he…

"Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.

AWS has a strong track record, a clear business model that isn’t predicated on gathering as much data as possible, and an awful lot to lose if they break their promises.

Lots of AI companies have some of these, but not to the same extent.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#210

Post by slekker »

Havoc wrote: post #194
trebligdivad wrote: post #52

It seems pretty impressive - I'm running it on my CPU (16 core AMD 3950x) and it's very very impressive at translation, and the image description is very impressive as well. I'm getting about 2.3token/s on it (compared to under 1/s on the Calme-3.2 I was previously using). It does tend to be a bit chatty unless you tell it not to be; pretty much everything it'll give you a 'breakdown' unless you tell it not to - so f…

The upcoming qwen3 series is supposed to be MoE...likely to give better tk/s on CPU

What's MoE?
Post reply on HN