How does it compare to CodeGemma for programming tasks?
Gemma 3 QAT Models: Bringing AI to Consumer GPUs
71–80 of 286 posts
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#72Earlier quoted context omitted.
https://huggingface.co/google/gemma-3-27b-it-qat-q4_0-gguf/t... > 17 days ago Anywaaay... I'm literally asking, quite honestly, if this is just an 'after the fact' update literally weeks later, that they uploaded a bunch of models, or if there is something more significant about this I'm missing.
Hi! Omar from the Gemma team here. Last time we only released the quantized GGUFs. Only llama.cpp users could use it (+ Ollama, but without vision). Now, we released the unquantized checkpoints, so anyone can quantize themselves and use in their favorite tools, including Ollama with vision, MLX, LM Studio, etc. MLX folks also found that the model worked decently with 3 bits compared to naive 3-bit, so by releasing th…
Hey Omar, is there any chance that Gemma 3 might get a speech (ASR/AST/TTS) release?
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#73Earlier quoted context omitted.
Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…
My tooling doesn't measure TPS yet. It feels snappy to me on MLX. I agree that hosted models are usually a better option for most people - much faster, higher quality, handle longer inputs, really cheap. I enjoy local models for research and for the occasional offline scenario. I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from…
>I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources.
Think it is NOT just you. Most company with decent management also would not want their data going to anything outside the physical server they have in control of. But yeah for most people just use an app and hosted server. But this is HN,there are ppl here hosting their own email servers, so shouldn't be too hard to run llm locally.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#74I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…
Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…
> Whereas on local LLM, I watch it painstakingly prints preambles that I don't care about, and get what I actually need after 20 seconds.
You may need to "right-size" the models you use to match your hardware, model, and TPS expectations, which may involve using a smaller version of the model with faster TPS, upgrading your jardware, or paying for hosted models.
Alternatively, if you can use agentic workflows or tools like Aider, you don't have to watch the model work slowly with large modles locally. Instead you queue work for it, go to sleep, or eat, or do other work, and then much later look over the Pull Requests whenever it completes them.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#75When I see 32B or 70B models performing similarly to 200+B models, I don’t know what to make of this. Either the latter contains more breadth of information but we have managed to distill latent capabilities to be similar, the larger models are just less efficient, or the tests are not very good.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#76Earlier quoted context omitted.
Why is sillytavern the only LLM frontend which matters?
I tried sillytavern a few weeks ago... wow, that is an "interesting" UI! I blundered around for a while, couldn't figure out how to do anything useful... and then installed LM Studio instead.
I personally thought the lorebook feature was quite neat and then quickly gave up on it because I couldn't get it to trigger, ever.
Whatever those keyword things are, they certainly don't seem to be doing any form of RAG.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#77This is what local LLMs need—being treated like first-class citizens by the companies that make them. That said, the first graph is misleading about the number of H100s required to run DeepSeek r1 at FP16. The model is FP8.
so what is the real comparison against DeepSeek r1 ? Would be good to know which is actually more cost efficient and open (reproducible build) to run locally.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#78I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…
Been super impressed with local models on mac. Love that the gemma models have 128k token context input size. However, outputs are usually pretty short Any tips on generating long output? Like multiple pages of a document, a story, a play or even a book?
This is basically the opposite of what I've experienced - at least compared to another recent entry like IBM's Granite 3.3.
By comparison, Gemma3's output (both 12b and 27b) seems to typically be more long/verbose, but not problematically so.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#79This is what local LLMs need—being treated like first-class citizens by the companies that make them. That said, the first graph is misleading about the number of H100s required to run DeepSeek r1 at FP16. The model is FP8.
so what is the real comparison against DeepSeek r1 ? Would be good to know which is actually more cost efficient and open (reproducible build) to run locally.
half the amount of those dots is what it takes. but also, why compare a 27B model with a +600B? that doesn't make sense.
Re: Gemma 3 QAT Models: Bringing AI to Consumer GPUs
#80Earlier quoted context omitted.
My tooling doesn't measure TPS yet. It feels snappy to me on MLX. I agree that hosted models are usually a better option for most people - much faster, higher quality, handle longer inputs, really cheap. I enjoy local models for research and for the occasional offline scenario. I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from…
>I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources. Think it is NOT just you. Most company with decent management also would not want their data going to anything outside the physical server they have in control of. But yeah for most people just use an app and hosted server. But this is HN,there are ppl he…
"Most company with decent management also would not want their data going to anything outside the physical server they have in control of."
I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.