They keep mentioning the RTX 3090 (with 24 GB VRAM), but the model is only 14.1 GB. Shouldn’t it fit a 5060 Ti 16GB, for instance?
Just checked - 19 gigs with 8k context @ q8 kv.Plus another 2.5-ish or so for OS etc.
...so yeah 3090
Gemma 3 QAT Models: Bringing AI to Consumer GPUs
191–200 of 286 posts
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#192Earlier quoted context omitted.
"Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.
Yeah, this has been confusing me a bit. I'm not complaining by ANY means, but why does it suddenly feel like everyone cares about data privacy in LLM contexts, way more than previous attitudes to allowing data to sit on a bunch of random SaaS products? I assume because of the assumption that the AI companies will train off of your data, causing it to leak? But I thought all these services had enterprise tiers where t…
It is due to the risk of a leak.
Laundering of data through training makes it a more complicated case than a simple data theft or copyright infringement.
Leaks could be accidental, e.g. due to an employee logging in to their free-as-in-labor personal account instead of a no-training Enterprise account. It's safer to have a complete ban on providers that may collect data for training.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#193Earlier quoted context omitted.
You can probably find some long-form tuned models on HF. I've had decent results with QwQ-32B (which I can run on my desktop) and Mistral Large (which I have to run on my server). Generating and refining an outline before writing the whole piece can help, and you can also split the piece up into multiple outputs (working a paragraph or two at a time, for instance). So far I've found it to be a tough process, with mix…
Thank you, will try out your suggestions Have you used something like a director model to supervise the output? If so, could you comment on the effectiveness of it and potentially any tips?
Nope, sounds neat though. There's so much to keep up with in this space.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#194It seems pretty impressive - I'm running it on my CPU (16 core AMD 3950x) and it's very very impressive at translation, and the image description is very impressive as well. I'm getting about 2.3token/s on it (compared to under 1/s on the Calme-3.2 I was previously using). It does tend to be a bit chatty unless you tell it not to be; pretty much everything it'll give you a 'breakdown' unless you tell it not to - so f…
The upcoming qwen3 series is supposed to be MoE...likely to give better tk/s on CPU
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#195Definitely my current fav. Also interesting that for many questions the response is very similar to the gemini series. Must be sharing training datasets pretty directly.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#196Anyone packaged one of these in an iPhone App? I am sure it is doable, but I am curious what tokens/sec is possible these days. I would love to ship "private" AI Apps if we can get reasonable tokens/sec.
If you ever ship a private AI app, don't forget to implement the export functionality, please!
You mean conversations?
Just the jsonl of the standard hf dataset format to import into other systems?
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#197Earlier quoted context omitted.
Yeah, this has been confusing me a bit. I'm not complaining by ANY means, but why does it suddenly feel like everyone cares about data privacy in LLM contexts, way more than previous attitudes to allowing data to sit on a bunch of random SaaS products? I assume because of the assumption that the AI companies will train off of your data, causing it to leak? But I thought all these services had enterprise tiers where t…
It is due to the risk of a leak. Laundering of data through training makes it a more complicated case than a simple data theft or copyright infringement. Leaks could be accidental, e.g. due to an employee logging in to their free-as-in-labor personal account instead of a no-training Enterprise account. It's safer to have a complete ban on providers that may collect data for training.
Their entire business model based on taking other peoples stuff. I cant imagine someone would willingly drown with the sinking ship if the entire cargo is filled with lifeboats - just because they promised they would.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#198Earlier quoted context omitted.
MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.
> MacBook Pro M2 with 64GB of RAM Are there non-mac options with similar capabilities?
I remember seeing a post about someone running the full size DeepSeek model in a dual-Xeon server with a ton of RAM.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#199Earlier quoted context omitted.
> MacBook Pro M2 with 64GB of RAM Are there non-mac options with similar capabilities?
Anything with the Radeon 8060S/Ryzen AI Max+ 395. One of the popular MiniPC Chinese brands has them for preorder[0] with shipping starting May 7th. Framework also has them, but shipping Q3. 0: https://www.gmktec.com/products/prepaid-deposit-amd-ryzen™-a...
I've never been able to get ROCm working reliably personally.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#200Earlier quoted context omitted.
I am only able to get the Gemma-3-27b-it-qat-Q4_0.gguf (15.6GB) to run with a 100 token context size on a 5070 ti (16GB) using llamacpp. Prompt Tokens: 10 Time: 229.089 ms Speed: 43.7 t/s Generation Tokens: 41 Time: 959.412 ms Speed: 42.7 t/s
I didn't realise the 5070 is slower than the 3090. Thanks. If you want a bit more context, try -ctv q8 -ctk q8 (from memory so look it up) to quant the kv cache. Also an imatrix gguf like iq4xs might be smaller with better quality
I answered the question directly. IQ4_X_S is smaller, but slower and less accurate than Q4_0. The parent comment specifically asked about the QAT version. That's literally what this thread is about. The context-length mention was relevant to show how it's only barely usable.