Gemma 3 QAT Models: Bringing AI to Consumer GPUs • Hacker News Board

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#221

Post by [deleted] » Mon, Apr 21, 2025, 6:19 AM UTC

DJHenk wrote: post #125

rs186 wrote: post #48

Earlier quoted context omitted.

Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…

> More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. There is another aspect to consider, aside from privacy. These models are trained by downloading every scrap of information from the…

[deleted]

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#222

Post by taneq » Mon, Apr 21, 2025, 6:39 AM UTC

rs186 wrote: post #155

overfeed wrote: post #74

Earlier quoted context omitted.

> Whereas on local LLM, I watch it painstakingly prints preambles that I don't care about, and get what I actually need after 20 seconds. You may need to "right-size" the models you use to match your hardware, model, and TPS expectations, which may involve using a smaller version of the model with faster TPS, upgrading your jardware, or paying for hosted models. Alternatively, if you can use agentic workflows or tool…

I have a 4070 super for gaming, and used it to play with LLM a few times. It is by no means a bad card, but I realize that unless I want to get 4090 or new Macs that I don't have any other use for, I can only use it to run smaller models. However, most smaller models aren't satisfactory and are still slower than hosted LLMs. I haven't found a model that I am happy with for my hardware. Regarding agentic workflows --…

As I understand it, these models are limited on GPU memory far more than GPU compute. You’d be better off with dual 4070s than with a single 4090 unless the 4090 has more RAM than the other two combined.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#223

Post by andhuman » Mon, Apr 21, 2025, 6:41 AM UTC

itake wrote: post #215

Samin100 wrote: post #147

I have a few private “vibe check” questions and the 4 bit QAT 27B model got them all correctly. I’m kind of shocked at the information density locked in just 13 GB of weights. If anyone at Deepmind is reading this — Gemma 3 27B is the single most impressive open source model I have ever used. Well done!

I tried to use the -it models for translation, but it completely failed at translating adult content. I think this means I either have to train the -pt model with my own instruction tuning or use another provider :(

Have you tried Mistral Small 24b?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#224

Post by manjunaths » Mon, Apr 21, 2025, 6:58 AM UTC

I am running this on 16 GB AMD Radeon 7900 GRE with 64 GB machine with ROCm and llama.cpp on Windows 11. I can use Open-webui or the native gui for the interface. It is made available via an internal IP to all members of my home.

It runs at around 26 tokens/sec and FP16, FP8 is not supported by the Radeon 7900 GRE.

I just love it.

For coding QwQ 32b is still king. But with a 16GB VRAM card it gives me ~3 tokens/sec, which is unusable.

I tried to make Gemma 3 write a powershell script with Terminal gui interface and it ran into dead-ends and finally gave up. QwQ 32B performed a lot better.

But for most general purposes it is great. My kid's been using it to feed his school textbooks and ask it questions. It is better than anything else currently.

Somehow it is more "uptight" than llama or the chinese models like Qwen. Can't put my finger on it, the Chinese models seem nicer and more talkative.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#225

Post by prometheon1 » Mon, Apr 21, 2025, 6:59 AM UTC

Der_Einzige wrote: post #8

emrah wrote: post #2

Available on ollama: https://ollama.com/library/gemma3

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they kne…

From the HN guidelines: https://news.ycombinator.com/newsguidelines.html

> Be kind. Don't be snarky.

> Please don't post shallow dismissals, especially of other people's work.

In my opinion, your comment is not in line with the guidelines. Especially the part about sillytavern being the only LLM frontend that matters. Telling the devs of any LLM frontend except sillytavern that their app doesn't matter seems exactly like a shallow dismissal of other people's work to me.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#226

Post by jychang » Mon, Apr 21, 2025, 7:32 AM UTC

mnoronha wrote: post #166

simonw wrote: post #162

Earlier quoted context omitted.

I expect not. On my Mac at least I've found I need a bunch of GB free to have anything else running at all.

Any idea why MLX and ollama use such different amounts of ram?

I don't think ollama is quantizing the embeddings table, which is still full FP16.

If you're using MLX, that means you're on a mac, in which case ollama actually isn't your best option. Either directly use llama.cpp if you're a power user, or use LM Studio if you want something a bit better than ollama but more user friendly than llama.cpp. (LM Studio has a GUI and is also more user friendly than ollama, but has the downsides of not being as scriptable. You win some, you lose some.)

Don't use MLX, it's not as fast/small as the best GGUFs currently (and also tends to be more buggy, it currently has some known bugs with japanese). Download the LM Studio version of the Gemma 3 QAT GGUF quants, which are made by Bartowski. Google actually directly mentions Bartowski in blog post linked above (ctrl-f his name), and his models are currently the best ones to use.

https://huggingface.co/bartowski/google_gemma-3-27b-it-qat-G...

The "best Gemma 3 27b model to download" crown has taken a very roundabout path. After the initial Google release, it went from Unsloth Q4_K_M, to Google QAT Q4_0, to stduhpf Q4_0_S, to Bartowski Q4_0 now.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#227

Post by jychang » Mon, Apr 21, 2025, 7:44 AM UTC

simonw wrote: post #64

tomrod wrote: post #60

Earlier quoted context omitted.

Simon, what is your local GPU setup? (No doubt you've covered this, but I'm not sure where to dig up).

MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.

MLX is slower than GGUFs on Macs.

On my M1 Max macbook pro, the GGUF version bartowski/google_gemma-3-27b-it-qat-GGUF is 15.6gb and runs at 17tok/sec, whereas mlx-community/gemma-3-27b-it-qat-4bit is 16.8gb and runs at 15tok/sec. Note that both of these are the new QAT 4bit quants.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#228

Post by jychang » Mon, Apr 21, 2025, 7:47 AM UTC

simonw wrote: post #120

littlestymaar wrote: post #115

Earlier quoted context omitted.

> and it only uses ~22Gb (via Ollama) or ~15GB (MLX) Why is the memory use different? Are you using different context size in both set-ups?

No idea. MLX is its own thing, optimized for Apple Silicon. Ollama uses GGUFs. https://ollama.com/library/gemma3:27b-it-qat says it's Q4_0. https://huggingface.co/mlx-community/gemma-3-27b-it-qat-4bit says it's 4bit. I think those are the same quantization?

Those are the same quant, but this is a good example of why you shouldn't use ollama. Either directly use llama.cpp, or use something like LM Studio if you want something with a GUI/easier user experience.

The Gemma 3 17b QAT GGUF should be taking up ~15gb, not 22gb.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#229

Post by jychang » Mon, Apr 21, 2025, 7:53 AM UTC

itake wrote: post #215

Samin100 wrote: post #147

I have a few private “vibe check” questions and the 4 bit QAT 27B model got them all correctly. I’m kind of shocked at the information density locked in just 13 GB of weights. If anyone at Deepmind is reading this — Gemma 3 27B is the single most impressive open source model I have ever used. Well done!

I tried to use the -it models for translation, but it completely failed at translating adult content. I think this means I either have to train the -pt model with my own instruction tuning or use another provider :(

Try mradermacher/amoral-gemma3-27B-v2-qat-GGUF

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#230

Post by ein0p » Mon, Apr 21, 2025, 8:07 AM UTC

starik36 wrote: post #178

ein0p wrote: post #127

Earlier quoted context omitted.

Sometimes TPS doesn't matter. I've generated textual descriptions for 100K or so images in my photo archive, some of which I have absolutely no interest in uploading to someone else's computer. This works pretty well with Gemma. I use local LLMs all the time for things where privacy is even remotely important. I estimate this constitutes easily a quarter of my LLM usage.

I was thinking of doing the same, but I would like to include people's name. in the description. For example "Jennifer looking out in the desert sky.". As it stands, Gemma will just say "Woman looking out in the desert sky."

Most search rankers do not consider word order, so if you could also append the person's name at the end of text description, it'd probably work well enough for retrieval and ranking at least.

If you want natural language to resolve the names, that'd at a minimum require bounding boxes of the faces and their corresponding names. It'd also require either preprocessing, or specialized training, or both. To my knowledge no locally-hostable model as of today has that. I don't know if any proprietary models can do this either, but it's certainly worth a try - they might just do it. The vast majority of the things they can do is emergent, meaning they were never specifically trained to do them.