Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma3 – The current strongest model that fits on a single GPU

ollama.com

Post reply on HN
Print view View original

121–130 of 148 posts

Re: Gemma3 – The current strongest model that fits on a single GPU

#121

Post by archerx »

_1 wrote: post #107
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

How are you grading these? Are you going on feeling, or do you have a formalized benchmarking process?

From just using them a lot and getting the results that I want without going "ugh!".

Re: Gemma3 – The current strongest model that fits on a single GPU

#122

Post by archerx »

dudefeliciano wrote: post #85
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

what hardware are you using those on? Is it still prohibitively expensive to self-host a model that gives decent outputs (sorry my last experience has been underwhelming with llama a while back)

I have an AMD 6700 XT card with 12gb of VRAM and a 24 core cpu with 48gigs of ram. This is the bare minimum,

Re: Gemma3 – The current strongest model that fits on a single GPU

#123

Post by nathanasmith »

amelius wrote: post #73
antirez wrote: post #22

After reading the technical report do the effort of downloading the model and run it against a few prompts. In 5 minutes you understand how broken LLM benchmarking is.

Aren't there any "blind" benchmarks?

Unfortunately that wouldn't help as much as you think since talented AI labs can just watch the public leaderboard and note what models move up and down to deduce and target whatever the hidden benchmark is testing.

Re: Gemma3 – The current strongest model that fits on a single GPU

#124

Post by archerx »

thom wrote: post #61
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Could you talk a little more about your D&D usage? This has turned into one of my primary use cases for ChatGPT, cooking up encounters or NPCs with a certain flavour if I don't have time to think something up myself. I've also been working on hooking up to the D&D Beyond API so you can get everything into homebrew monsters and encounters.

I noticed the prompt makes a big difference in the experience you get. I have a simple game prompt.

The first prompt I tested out I got from this video; https://www.youtube.com/watch?v=0Cq-LuJnaRg

It was ok and produces shallow adventures.

The second one I tried was from this site; https://www.rpgprompts.com/post/dungeons-dragons-chatgpt-pro...

a bit better and is easier to modify but still shallow.

The best one I have tried so far is this one from reddit; https://old.reddit.com/r/ChatGPT/comments/zoiqro/most_improv...

It is a super long prompt and I had to edit it a lot, and manually extract the data from some of the links but it has been the best experience by far. I even became "friends" with an NPC who accompanied me on a quest and it was a lot of fun and I was fully engaged.

The model of choice matters but even llama 1B and 2B can handle some stories.

Re: Gemma3 – The current strongest model that fits on a single GPU

#125

Post by tarruda »

whbrown wrote: post #116
tarruda wrote: post #92

My usual non-scientific benchmark is asking it to implement the game Tetris in python, and then iterating with the LLM to fix/tweak it. My prompt to Gemma 27b (q4) on open webui + ollama: "Can you create the game tetris in python?" It immediately starts writing code. After the code is finished, I noticed something very strange, it starts a paragraph like this: " Key improvements and explanations: Clearer Code Structu…

Those sound like the sort of issues which could be caused by your server silently truncating the middle of your prompts. By default, Ollama uses a context window size of 2048 tokens.

I checked this, the whole conversation was about 1000 tokens.

I suspect the Ollama version might have wrong default settings, such as conversation delimiters. The experience of Gemma 3 in AI studio is completely different.

Re: Gemma3 – The current strongest model that fits on a single GPU

#126

Post by tarruda »

anon373839 wrote: post #63
antirez wrote: post #26

Earlier quoted context omitted.

The model performs very poorly in practice, while in the benchmark it is shown to be DeepSeek V3 level. It's not terrible but it's at another level compared to the models it is very close to (a bit better / a bit worse) in the benchmarks.

I’d recommend trying it on Google AI Studio (aistudio.google.com). I am getting exceptional results on a handful of novel problems that require deep domain knowledge and structured reasoning. I’m not able to replicate this performance with Ollama, so I suspect something is a bit off.

Same experience here: On AI Studio, this is easily one of the strongest models I have used, including when compared to proprietary LLMs.

But ollama and openwebui performance is very bad, even when running the FP16 version. I also tried to mirror some of AI studio settings (temp 1 and top p 0.95) but couldn't get it to produce anything useful.

I suspect there's some bug in the ollama releases (possibly wrong conversation delimiters?). If this is fixed, I will definitely start using Gemma 3 27b as my main model.

Re: Gemma3 – The current strongest model that fits on a single GPU

#127

Post by tarruda »

pzo wrote: post #94
anon373839 wrote: post #63

Earlier quoted context omitted.

I’d recommend trying it on Google AI Studio (aistudio.google.com). I am getting exceptional results on a handful of novel problems that require deep domain knowledge and structured reasoning. I’m not able to replicate this performance with Ollama, so I suspect something is a bit off.

Maybe model is sensitive to quantization, by default ollama quantize it significantly.

I tried ollama fp16 and it had the same issues.

Re: Gemma3 – The current strongest model that fits on a single GPU

#128

Post by tarruda »

alekandreev wrote: post #51
antirez wrote: post #26

Earlier quoted context omitted.

The model performs very poorly in practice, while in the benchmark it is shown to be DeepSeek V3 level. It's not terrible but it's at another level compared to the models it is very close to (a bit better / a bit worse) in the benchmarks.

Hey, Gemma engineer here. Can you please share reports on the type of prompts and the implementation you used?

Can you share the all the recommended settings to run this LLM? It is clear that the performance is very good when running on AI studio. If possible, I'd like to use the all the same settings (temp, top-k, top-p, etc) on Ollama. AI studio only shows Temperature, top-p and output length.

Re: Gemma3 – The current strongest model that fits on a single GPU

#129

Post by tarruda »

tarruda wrote: post #92

My usual non-scientific benchmark is asking it to implement the game Tetris in python, and then iterating with the LLM to fix/tweak it. My prompt to Gemma 27b (q4) on open webui + ollama: "Can you create the game tetris in python?" It immediately starts writing code. After the code is finished, I noticed something very strange, it starts a paragraph like this: " Key improvements and explanations: Clearer Code Structu…

I ran the same prompt on google AI studio it had the same behavior of talking about improvements as if the code it wrote was not the first version.

Other than that, the experience was completely different:

- The game worked on first try

- I iterated with the model making enhancements. The first version worked but didn't show scores, levels or next piece, so I asked it to implement those features. It then produced a new version which almost worked: The only problem was that levels were increasing whenever a piece fell, and I didn't notice any increase in falling speed.

- So I reported the problems with level tracking and falling speed and it produced a new version which crashed immediately. I pasted the error and it was able to fix it in the next version

- I kept iterating with the model, fixing issues until it finally produced a perfectly working tetris game which I played and eventually lost due to high falling speed.

- As a final request, I asked it to port the latest working version of the game to JS/HTML with the implementation self contained in a file. It produced a broken implementation, but I was able to fix it after tweaking it a little bit.

Gemma 3 27b on Google AI studio is easily one of the best LLMs I've used for coding.

Unfortuantely I can't seem to reproduce the same results in ollama/open webui, even when running the full fp16 version.

Re: Gemma3 – The current strongest model that fits on a single GPU

#130

Post by memhole »

archerx wrote: post #47
memhole wrote: post #35

Earlier quoted context omitted.

Do you mostly stick with smaller models? I’m pretty surprised at how good the smaller models can be at times now. A year ago they were nearly useless. I kind of like too the hallucinations are more obvious sometimes. Or at least it seems like they are.

I like the smaller models because they are faster. I even got a Llama 3 1B model running on TinkerBoard 2S and it was fun to play around with and not too slow. The smaller models are still good at summarizing and other basic tasks. For coding they start showing their limits but still work great for trying to figure out issues in small bits of code. The real issue with local models is managing context. smaller models…

Yeah, that’s why I like the smaller models too. Big context windows and intelligent enough most of the time. They don’t follow instructions as well as the larger models ime. But then on the flip side the reasoning models struggle to deviate. I gave deepseek an existential crisis by accident the other day lol.

Agreed on personalities. Phi, I think because of the curated training data comes across as very dry.

Post reply on HN