[deleted]
Gemma 3 QAT Models: Bringing AI to Consumer GPUs
281–286 of 286 posts
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#282Earlier quoted context omitted.
What kind of functionality do you need from the model? For basic conversation and RAG, you can use tinyllama or qwen-2.5-0.5b, both of which run on a raspberry pi at around 5-20 tokens per second
I am looking for structured output at about 100-200 tokens/second on iPhone 14+. Any pointers?
The qwq-2.5-0.5b is the tiniest useful model I've used, and pretty easy to fine-tune locally on a Mac. Haven't tried it on an iPhone, but given it runs at about 150-200 tokens/second on a Mac, I'm kinda doubtful it could do the same on an iPhone. But I guess you'd just have to try
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#283Earlier quoted context omitted.
On a M4 Max 128GB via LM Studio: query: "make me a snake game in python with pygame" (mlx 4 bit quant) mlx-community/gemma-3-27b-it-qat@4bit: 26.39 tok/sec • 1681 tokens 0.63s to first token (gguf 4 bit quant) lmstudio-community/gemma-3-27b-it-qat: 22.72 tok/sec • 1866 tokens 0.49s to first token using Unsloth's settings: https://docs.unsloth.ai/basics/tutorial-how-to-run-and-fine-...
I genuinely would have expected a $3,500+ setup to do better than just 10x pure-CPU on a AMD Ryzen 9 8945HS.
Find another laptop that does that well.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#284Earlier quoted context omitted.
Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…
> Can you quote tps? LLM Studio running on a Mac Studio M4 Max with 128GB, gemma-3-27B-it-QAT-Q4_0.gguf with a 4096 token context I get 8.89 tps.
Gah, turns out I was running the Mac in low power mode!
I get 24tps in LM Studio now with gemma-3-27b-it-qat.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#285Earlier quoted context omitted.
> Can you quote tps? LLM Studio running on a Mac Studio M4 Max with 128GB, gemma-3-27B-it-QAT-Q4_0.gguf with a 4096 token context I get 8.89 tps.
That's pretty terrible. I'm getting 18tok/sec Gemma 3 27b QAT on a M1 Max 32gb macbook.
I was running the mac in low power mode!!! Getting 24tps now.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#286Earlier quoted context omitted.
The original gemma3:27b also took only 22GB using Ollama on my 64GB MacBook. I'm quite confused that the QAT took the same. Do you know why? Which model is better? `gemma3:27b`, or `gemma3:27b-qat`?
Look up 27b in https://ollama.com/library/gemma3/tags You'll find the id a418f5838eaf which also corresponds to 27b-it-q4_K_M
Just following this comment up as a note-to-self: just as `kgwgk noted, the default gemma3:27B model has ID a418f5838eaf, which corresponds to 27b-it-q4_K_M. But the new gemma3:27B quantization-aware training (QAT) model being discussed is gemma3:27b-it-qat with ID 29eb0b9aeda3.