Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

developers.googleblog.com

Post reply on HN
Print view View original

251–260 of 286 posts

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#251

Post by iamnotagenius »

terhechte wrote: post #87
simonw wrote: post #80

Earlier quoted context omitted.

"Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.

Or GitHub. I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub. All big llm providers offer no-training-on-your-data business plans.

[dead]

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#252

Post by iamnotagenius »

HachiWari8 wrote: post #183
perching_aix wrote: post #70

This is my first time trying to locally host a model - gave both the 12B and 27B QAT models a shot. I was both impressed and disappointed. Setup was piss easy, and the models are great conversationalists. I have a 12 gig card available and the 12B model ran very nice and swift. However, they're seemingly terrible at actually assisting with stuff. Tried something very basic: asked for a powershell one liner to get the…

I tried the 27B QAT model and it hallucinates like crazy. When I ask it for information about some made up person, restaurant, place name, etc., it never says "I don't know about that" and instead seems eager to just make up details. The larger local models like the older Llama 3.3 70B seem better at this, but are also too big to fit on a 24GB GPU.

[dead]

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#253

Post by codybontecou »

simonw wrote: post #174
codybontecou wrote: post #172

Earlier quoted context omitted.

Can you run the mlx-variation of this model through Ollama so that I can interact with it in Open WebUI?

I haven't tried it yet but there's an MLX project that exposes an OpenAI-compatible serving endpoint that should work with Open WebUI: https://github.com/madroidmaq/mlx-omni-server

Appreciate the link. I'll try to tinker with it later today.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#254

Post by dockerd »

hskalin wrote: post #35
holografix wrote: post #3

Could 16gb vram be enough for the 27b QAT version?

With ollama you could offload a few layers to cpu if they don't fit in the VRAM. This will cost some performance ofcourse but it's much better than the alternative (everything on cpu)

Does it work on LM Studio? Loading 27b-it-qat taking up more than 22GB on 24GB mac.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#255

Post by iAMkenough »

Der_Einzige wrote: post #105
Zambyte wrote: post #25

Earlier quoted context omitted.

The significant convenience benefits outweigh the higher TPS that vLLM offers in the context of my single machine homelab GPU server. If I was hosting it for something more critical than just myself and a few friends chatting with it, sure. Being able to just paste a model name into Open WebUI and run it is important to me though. It is important to know about both to decide between the two for your use case though.

Running any HF model on vllm is as simple as pasting a model name into one command in your terminal.

Had to build it from source to run on my Mac, and the experimental support doesn't seem to include these latest Gemma 3 QAT models on Apple Silicon.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#256

Post by terhechte »

perching_aix wrote: post #96
terhechte wrote: post #90

Earlier quoted context omitted.

Local models, due to their size more than big cloud models, favor popular languages rather than more niche ones. They work fantastic for JavaScript, Python, Bash but much worse at less popular things like Clojure, Nim or Haskell. Powershell is probably on the less popular side compared to Js or Bash. If this is your main use case you can always try to fine tune a model. I maintain a small llm bench of different progr…

How accessible and viable is model fine-tuning? I'm not in the loop at all unfortunately.

This is a very accessible way of playing around with the topic: https://transformerlab.ai

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#257

Post by aukejw »

simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

[dead]

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#259

Post by phaedrix »

jychang wrote: post #227
simonw wrote: post #64

Earlier quoted context omitted.

MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.

MLX is slower than GGUFs on Macs. On my M1 Max macbook pro, the GGUF version bartowski/google_gemma-3-27b-it-qat-GGUF is 15.6gb and runs at 17tok/sec, whereas mlx-community/gemma-3-27b-it-qat-4bit is 16.8gb and runs at 15tok/sec. Note that both of these are the new QAT 4bit quants.

No, in general mlx versions are always faster, ice tested most of them.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#260

Post by ygreif »

simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

Do many consumer GPUs have >20 gigabytes RAM? That sounds like a lot to me
Post reply on HN