Gemma 3 QAT Models: Bringing AI to Consumer GPUs • Hacker News Board

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#111

Post by simonw » Sun, Apr 20, 2025, 5:13 PM UTC

Elucalidavah wrote: post #108

simonw wrote: post #64

Earlier quoted context omitted.

MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.

> MacBook Pro M2 with 64GB of RAM Are there non-mac options with similar capabilities?

Yes, but I don't really know anything about those. https://www.reddit.com/r/LocalLLaMA/ is full of people running models on PCs with NVIDIA cards.

The unique benefit of an Apple Silicon Mac at the moment is that the 64GB of RAM is available to both the GPU and the CPU at once. With other hardware you usually need dedicated separate VRAM for the GPU.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#112

Post by simonw » Sun, Apr 20, 2025, 5:15 PM UTC

briandear wrote: post #109

The normal Gemma models seem to work fine on Apple silicon with Metal. Am I missing something?

These new special editions of those models claim to work better with less memory.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#113

Post by Casteil » Sun, Apr 20, 2025, 5:31 PM UTC

nolist_policy wrote: post #104

justanotheratom wrote: post #49

Anyone packaged one of these in an iPhone App? I am sure it is doable, but I am curious what tokens/sec is possible these days. I would love to ship "private" AI Apps if we can get reasonable tokens/sec.

FWIW, I can run Gemma-3-12b-it-qat on my Galaxy Fold 4 with 12Gb ram at around 1.5 tokens / s. I use plain llama.cpp with Termux.

Does this turn your phone into a personal space heater too?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#114

Post by porphyra » Sun, Apr 20, 2025, 5:31 PM UTC

It is funny that Microsoft had been peddling "AI PCs" and Apple had been peddling "made for Apple Intelligence" for a while now, when in fact usable models for consumer GPUs are only barely starting to be a thing on extremely high end GPUs like the 3090.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#115

Post by littlestymaar » Sun, Apr 20, 2025, 5:47 PM UTC

simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

> and it only uses ~22Gb (via Ollama) or ~15GB (MLX)

Why is the memory use different? Are you using different context size in both set-ups?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#116

Post by nico » Sun, Apr 20, 2025, 5:52 PM UTC

Casteil wrote: post #78

nico wrote: post #59

Earlier quoted context omitted.

Been super impressed with local models on mac. Love that the gemma models have 128k token context input size. However, outputs are usually pretty short Any tips on generating long output? Like multiple pages of a document, a story, a play or even a book?

This is basically the opposite of what I've experienced - at least compared to another recent entry like IBM's Granite 3.3. By comparison, Gemma3's output (both 12b and 27b) seems to typically be more long/verbose, but not problematically so.

I agree with you. The outputs are usually good, it’s just that for the use case I have now (writing several pages of long dialogs), the output is not as long as I’d want it, and definitely not as long as it’s supposedly capable of doing

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#117

Post by GaunterODimm » Sun, Apr 20, 2025, 5:55 PM UTC

rob_c wrote: post #32

Given how long between this being released and this community picking up on it... Lol

2days :/...

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#118

Post by claiir » Sun, Apr 20, 2025, 6:06 PM UTC

diggan wrote: post #5

First graph is a comparison of the "Elo Score" while using "native" BF16 precision in various models, second graph is comparing VRAM usage between native BF16 precision and their QAT models, but since this method is about doing quantization while also maintaining quality, isn't the obvious graph of comparing the quality between BF16 and QAT missing? The text doesn't seem to talk about it either, yet it's basically th…

Yea they mention a “perplexity drop” relative to naive quantization, but that’s meaningless to me. > We reduce the perplexity drop by 54% (using llama.cpp perplexity evaluation) when quantizing down to Q4_0.

Wish they showed benchmarks / added quantized versions to the arena! :>

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#119

Post by rob_c » Sun, Apr 20, 2025, 6:09 PM UTC

GaunterODimm wrote: post #117

rob_c wrote: post #32

Given how long between this being released and this community picking up on it... Lol

2days :/...

Given I know people running gemma3 on local devices for over almost a month now this is either a very slow news day or evidence of finger missing the pulse... https://blog.google/technology/developers/gemma-3/

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#120

Post by simonw » Sun, Apr 20, 2025, 6:13 PM UTC

littlestymaar wrote: post #115

simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

> and it only uses ~22Gb (via Ollama) or ~15GB (MLX) Why is the memory use different? Are you using different context size in both set-ups?

No idea. MLX is its own thing, optimized for Apple Silicon. Ollama uses GGUFs.

https://ollama.com/library/gemma3:27b-it-qat says it's Q4_0. https://huggingface.co/mlx-community/gemma-3-27b-it-qat-4bit says it's 4bit. I think those are the same quantization?