Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma 3 QAT Models: Bringing AI to Consumer GPUs

developers.googleblog.com

Post reply on HN
Print view View original

101–110 of 286 posts

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#101

Post by Terretta »

tarruda wrote: post #98
terhechte wrote: post #87

Earlier quoted context omitted.

Or GitHub. I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub. All big llm providers offer no-training-on-your-data business plans.

> I’m always amused when people don’t want to send fractions of their code to a LLM but happily host it on GitHub What amuses me even more is people thinking their code is too unique and precious, and that GitHub/Microsoft wants to steal it.

Unlikely they think Microsoft or GitHub wants to steal it.

With LLMs, they're thinking of examples that regurgitated proprietary code, and contrary to everyday general observation, valuable proprietary code does exist.

But with GitHub, the thinking is generally the opposite: the worry is that the code is terrible, and seeing it would be like giant blinkenlights* indicating the way in.

* https://en.wikipedia.org/wiki/Blinkenlights

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#102

Post by otabdeveloper4 »

rs186 wrote: post #48
simonw wrote: post #45

I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…

Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…

The only actually useful application of LLM's is processing large amounts of data for classification and/or summarizing purposes.

That's not the stuff you want to send to a public API, this is something you want as a 24/7 locally running batch job.

("AI assistant" is an evolutionary dead end, and Star Trek be damned.)

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#103

Post by [deleted] »

Alifatisk wrote: post #51
justanotheratom wrote: post #49

Anyone packaged one of these in an iPhone App? I am sure it is doable, but I am curious what tokens/sec is possible these days. I would love to ship "private" AI Apps if we can get reasonable tokens/sec.

If you ever ship a private AI app, don't forget to implement the export functionality, please!

[deleted]

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#104

Post by nolist_policy »

justanotheratom wrote: post #49

Anyone packaged one of these in an iPhone App? I am sure it is doable, but I am curious what tokens/sec is possible these days. I would love to ship "private" AI Apps if we can get reasonable tokens/sec.

FWIW, I can run Gemma-3-12b-it-qat on my Galaxy Fold 4 with 12Gb ram at around 1.5 tokens / s. I use plain llama.cpp with Termux.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#105

Post by Der_Einzige »

Zambyte wrote: post #25
Der_Einzige wrote: post #8

Earlier quoted context omitted.

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they kne…

The significant convenience benefits outweigh the higher TPS that vLLM offers in the context of my single machine homelab GPU server. If I was hosting it for something more critical than just myself and a few friends chatting with it, sure. Being able to just paste a model name into Open WebUI and run it is important to me though. It is important to know about both to decide between the two for your use case though.

Running any HF model on vllm is as simple as pasting a model name into one command in your terminal.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#106

Post by Der_Einzige »

mitjam wrote: post #95
simonw wrote: post #40

Earlier quoted context omitted.

Last I looked vLLM didn't work on a Mac.

Afaik vllm is for concurrent serving with batched inference for higher throughput, not single-user inference. I doubt inference throughput is higher with single prompts at a time than Ollama. Update: this is a good Intro to continuous batching in llm inference: https://www.anyscale.com/blog/continuous-batching-llm-infere...

It is much faster on single prompts than ollama. 3X is not unheard of

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#107

Post by Der_Einzige »

oezi wrote: post #33
Der_Einzige wrote: post #8

Earlier quoted context omitted.

How many times do I have to say this? Ollama, llamacpp, and many other projects are slower than vLLM/sglang. vLLM is a much superior inference engine and is fully supported by the only LLM frontends that matter (sillytavern). The community getting obsessed with Ollama has done huge damage to the field, as it's ineffecient compared to vLLM. Many people can get far more tok/s than they think they could if only they kne…

Why is sillytavern the only LLM frontend which matters?

It supports more sampler and other settings than anyone else.

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#108

Post by Elucalidavah »

simonw wrote: post #64
tomrod wrote: post #60

Earlier quoted context omitted.

Simon, what is your local GPU setup? (No doubt you've covered this, but I'm not sure where to dig up).

MacBook Pro M2 with 64GB of RAM. That's why I tend to be limited to Ollama and MLX - stuff that requires NVIDIA doesn't work for me locally.

> MacBook Pro M2 with 64GB of RAM

Are there non-mac options with similar capabilities?

Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs

#110

Post by simonw »

__float wrote: post #93
simonw wrote: post #80

Earlier quoted context omitted.

"Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.

While none of that is false, I think there's a big difference from shipping your data to an external LLM API and using AWS. Using AWS is basically a "physical server they have control of".

That's why AWS Bedrock and Google Vertex AI and Azure AI model inference exist - they're all hosted LLM services that offer the same compliance guarantees that you get from regular AWS-style hosting agreements.
Post reply on HN