Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma3 – The current strongest model that fits on a single GPU

ollama.com

Post reply on HN
Print view View original

141–148 of 148 posts

Re: Gemma3 – The current strongest model that fits on a single GPU

#141

Post by tarruda »

ac29 wrote: post #139
tarruda wrote: post #135

Earlier quoted context omitted.

Thanks for this, but I'm still unable to reproduce the results from Google AI studio. I tried your version and when I ask it to create a tetris game in python, the resulting file has syntax errors. I see strange things like a space in the middle of a variable name/reference or weird spacing in the code output.

Some models are more sensitive to quantization than others, presumably AI Studio is running the full 16 bit model. Try maybe the 8bit quant if you have the hardware for it? ollama run hf.co/unsloth/gemma-3-27b-it-GGUF:Q8_0

I tested the full fp16 gguf

Re: Gemma3 – The current strongest model that fits on a single GPU

#142

Post by vessenes »

danielhanchen wrote: post #87

I wrote a mini guide on running Gemma 3 at https://docs.unsloth.ai/basics/tutorial-how-to-run-gemma-3-e... ! The recommended settings according to the Gemma team are: temperature = 0.95 top_p = 0.95 top_k = 64 Also beware of double BOS tokens! You can run my uploaded GGUFs with the recommended chat template and settings via ollama run hf.co/unsloth/gemma-3-27b-it-GGUF:Q4_K_M

Daniel, as always, thanks for these. I had good results with your Q4_K_M quant on mac / llama.cpp. However, on Linux/A100/ollama, there is something very wrong with your Q8_0 quant. python code has indentation errors, missing close parens, quite a lot that's bad. I ran both with your suggested command lines, but of course could have been some mistake I made. I'm testing the bf16 on the A100 now to make sure it's not a hardware issue, but my gut is there's a model or ollama sampling problem here.

EDIT: 27b size

Re: Gemma3 – The current strongest model that fits on a single GPU

#143

Post by svachalek »

danielhanchen wrote: post #87

I wrote a mini guide on running Gemma 3 at https://docs.unsloth.ai/basics/tutorial-how-to-run-gemma-3-e... ! The recommended settings according to the Gemma team are: temperature = 0.95 top_p = 0.95 top_k = 64 Also beware of double BOS tokens! You can run my uploaded GGUFs with the recommended chat template and settings via ollama run hf.co/unsloth/gemma-3-27b-it-GGUF:Q4_K_M

This seems worse than the official Ollama build. First question I tried:

>>> who is president

The বর্তমানpresident of the United States is Джо Байден (JoeBiden).

Re: Gemma3 – The current strongest model that fits on a single GPU

#144

Post by sliken »

dudefeliciano wrote: post #85
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

what hardware are you using those on? Is it still prohibitively expensive to self-host a model that gives decent outputs (sorry my last experience has been underwhelming with llama a while back)

I'm tinkering with gemma 3 27B on a last gen 12 core ryzen. I get 5 tokens/sec.

Re: Gemma3 – The current strongest model that fits on a single GPU

#145

Post by kamranjon »

antirez wrote: post #26
toinewx wrote: post #24

Earlier quoted context omitted.

can you expand a bit?

The model performs very poorly in practice, while in the benchmark it is shown to be DeepSeek V3 level. It's not terrible but it's at another level compared to the models it is very close to (a bit better / a bit worse) in the benchmarks.

I really respect the work that you've done, but I am always very surprised when people just speak anecdotally as though it is truth with regards to AI models these days. It's as if everyone believes they are an expert now, but have nothing of substance to provide but their gut feelings.

It's as if people don't realize that these models are used for many different purposes, and subjectively one person could think one model is amazing and another person think it's awful. I just would hope that we could at least back up statements like "The model performs very poorly in practice" with actual data or at least some explanation of how it performed poorly.

Re: Gemma3 – The current strongest model that fits on a single GPU

#146

Post by PKop »

smcleod wrote: post #69

No mention of how well it's claimed to perform with tool calling? The Gemma series of models has historically been pretty poor when it comes to coding and tool calling - two things that are very important to agentic systems, so it will be interesting to see how 3 does in this regard.

I wasn't able to get function calls to work for Gemma3 in ollama, nor were others[0]. What is another way to run these models locally?

[0] https://github.com/ollama/ollama/issues/9680

[1] https://github.com/ollama/ollama/issues/9680#issuecomment-27...

Re: Gemma3 – The current strongest model that fits on a single GPU

#147

Post by janwas »

moffkalast wrote: post #70
hnfong wrote: post #62

Earlier quoted context omitted.

Quants at 4 bits are generally considered good, and 8 bits are generally considered overkill unless somehow need to squeeze the last bits of performance (in terms of generation quality) from it. There are papers to that effect though admittedly perhaps specific models might have divergent behavior ( https://arxiv.org/abs/2212.09720 ) All the above is subjective so maybe that’s true for you, but claiming there’s a lac…

> Quants at 4 bits are generally considered good, and 8 bits are generally considered overkill Two year old info, only really applies to heavily undertrained models with short tokenizers. Perplexity scores are a really terrible metric for measuring quantization impact, and quantized models tend to also score higher than they should in benchmarks ran as topk=1 where the added randomness seems to help. In my experience…

Strongly agree with the first part of your post :) BTW in addition to the weights, it's also interesting to consider the precision of accumulation. f16 is just not enough for the large matrix sizes we are now seeing.

(Gemma.cpp TL here) FYI we are a research testbed, not full-featured nor user-centric. Some interesting things there are the fp8 weights and extremely fast matmul especially on workstation CPUs, plus some attention to numerics.

Re: Gemma3 – The current strongest model that fits on a single GPU

#148

Post by anon373839 »

tarruda wrote: post #126
anon373839 wrote: post #63

Earlier quoted context omitted.

I’d recommend trying it on Google AI Studio (aistudio.google.com). I am getting exceptional results on a handful of novel problems that require deep domain knowledge and structured reasoning. I’m not able to replicate this performance with Ollama, so I suspect something is a bit off.

Same experience here: On AI Studio, this is easily one of the strongest models I have used, including when compared to proprietary LLMs. But ollama and openwebui performance is very bad, even when running the FP16 version. I also tried to mirror some of AI studio settings (temp 1 and top p 0.95) but couldn't get it to produce anything useful. I suspect there's some bug in the ollama releases (possibly wrong conversat…

Update: Unsloth is recommending a temperature of 0.1, not 1.0, if using Ollama. I don’t know why Ollama would require a 10x lower value, but it definitely helped. I also read some speculation that there might be an issue with the tokenizer.
Post reply on HN