Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

developers.googleblog.com

Post reply on HN
Print view View original

231–240 of 286 posts

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#231

Post by nolist_policy »

tbocek wrote: post #143
parched99 wrote: post #62

Earlier quoted context omitted.

I am only able to get the Gemma-3-27b-it-qat-Q4_0.gguf (15.6GB) to run with a 100 token context size on a 5070 ti (16GB) using llamacpp. Prompt Tokens: 10 Time: 229.089 ms Speed: 43.7 t/s Generation Tokens: 41 Time: 959.412 ms Speed: 42.7 t/s

This is probably due to this: https://github.com/ggml-org/llama.cpp/issues/12637 . This GitHub issue is about interleaved sliding window attention (iSWA) not available in llama.cpp for Gemma 3. This could reduce the memory requirements a lot. They mentioned for a certain scenario, going from 62GB to 10GB.

Ollama supports iSWA.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#233

Post by Havoc »

slekker wrote: post #210
Havoc wrote: post #194

Earlier quoted context omitted.

The upcoming qwen3 series is supposed to be MoE...likely to give better tk/s on CPU

What's MoE?

Mixture of experts like other guy said - everything gets loaded into mem but not every byte is needed to generate a token (unlike classic LLMs like gemma).

So for devices that have lots of mem but weaker processing power it can get you similar output quality but faster. So tends to do better on CPU and APU like setups

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#234

Post by freeamz »

ipdashc wrote: post #181
simonw wrote: post #80

Earlier quoted context omitted.

"Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.

Yeah, this has been confusing me a bit. I'm not complaining by ANY means, but why does it suddenly feel like everyone cares about data privacy in LLM contexts, way more than previous attitudes to allowing data to sit on a bunch of random SaaS products? I assume because of the assumption that the AI companies will train off of your data, causing it to leak? But I thought all these services had enterprise tiers where t…

In Scandinavian financial related severs must in the country! That always sounded like a sane approach. The whole putting your data on saas or AWS just seems like the same "Let's shift the responsibility to a big player".

Any important data should NOT be in devices that is NOT physically with in our jurisdiction.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#235

Post by retinaros »

rafaelmn wrote: post #163
999900000999 wrote: post #94

Assuming this can match Claude's latest, and full time usage ( as in you have a system that's constantly running code without any user input,) you'd probably save 600 to 700 a month. A 4090 is only 2K and you'll see an ROI within 90 days. I can imagine this will serve to drive prices for hosted llms lower. At this level any company that produces even a nominal amount of code should be running LMS on prem( AWS if your…

I'd say using a Mac studio with M4 Max and 128 GB RAM will get you way further than 4090 in context size and model size. Cheaper than 2x4090 and less power while being a great overall machine. I think these consumer GPUs are way too expensive for the amount of memory they pack - and that's intentional price discrimination. Also the builds are gimmicky. It's just not setup for AI models, and the versions that are cost…

two 3090 are the way to go

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#236

Post by Tsarp »

mark_l_watson wrote: post #134

Indeed!! I have swapped out qwen2.5 for gemma3:27b-it-qat using Ollama for routine work on my 32G memory Mac. gemma3:27b-it-qat with open-codex, running locally, is just amazingly useful, not only for Python dev, but for Haskell and Common Lisp also. I still like Gemini 2.5 Pro and o3 for brainstorming or working on difficult problems, but for routine work it (simply) makes me feel good to have everything open source…

What tps are you hitting? And did you have to change KV size?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#237

Post by nxobject »

mark_l_watson wrote: post #134

Indeed!! I have swapped out qwen2.5 for gemma3:27b-it-qat using Ollama for routine work on my 32G memory Mac. gemma3:27b-it-qat with open-codex, running locally, is just amazingly useful, not only for Python dev, but for Haskell and Common Lisp also. I still like Gemini 2.5 Pro and o3 for brainstorming or working on difficult problems, but for routine work it (simply) makes me feel good to have everything open source…

Fellow owner of a 32GB MBP here: how much memory does it use while resident - or, if swapping happens, do you see the effects in your day to day work? I’m in the awkward position of using on a daily basis a lot of virtualized bloated Windows software (mostly SAS).

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#238

Post by Alifatisk »

idonotknowwhy wrote: post #196
Alifatisk wrote: post #51

Earlier quoted context omitted.

If you ever ship a private AI app, don't forget to implement the export functionality, please!

You mean conversations? Just the jsonl of the standard hf dataset format to import into other systems?

Yeah I mean conversations.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#239

Post by itake »

andhuman wrote: post #223
itake wrote: post #215

Earlier quoted context omitted.

I tried to use the -it models for translation, but it completely failed at translating adult content. I think this means I either have to train the -pt model with my own instruction tuning or use another provider :(

Have you tried Mistral Small 24b?

My current architecture is an on-device model for fast translation and then replace that with a slow translation (via an API call) when its ready.

24b would be too small to run on device and I'm trying to keep my cloud costs low (meaning I can't afford to host a small 24b 24/7).

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#240

Post by itake »

jychang wrote: post #229
itake wrote: post #215

Earlier quoted context omitted.

I tried to use the -it models for translation, but it completely failed at translating adult content. I think this means I either have to train the -pt model with my own instruction tuning or use another provider :(

Try mradermacher/amoral-gemma3-27B-v2-qat-GGUF

My current architecture is an on-device model for fast translation and then replace that with a slow translation (via an API call) when its ready.

24b would be too small to run on device and I'm trying to keep my cloud costs low (meaning I can't afford to host a small 27b 24/7).

Post reply on HN