Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma3 – The current strongest model that fits on a single GPU

ollama.com

Post reply on HN
Print view View original

21–30 of 148 posts

Re: Gemma3 – The current strongest model that fits on a single GPU

#21

Post by blooalien »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

I have had nothing but good results using the Qwen2.5 and Hermes3 models. The response times and token generation speeds have been pretty fantastic compared against other models I've tried, too.

Re: Gemma3 – The current strongest model that fits on a single GPU

#23

Post by squeakywhite »

mythz wrote: post #3

Not sure if anyone else experiences this, but ollama downloads starts off strong but the last few MBs take forever. Finally just finished downloading (gemma3:27b). Requires the latest version of Ollama to use, but now working, getting about 21 tok/s on my local 2x A4000. From my few test prompts looks like a quality model, going to run more tests to compare against mistral-small:24b to see if it's going to become my…

I experienced this just now. The download slowed down to approx 500kB/s for the last 1% or so. When this happens, you can Ctrl+C to cancel and then start the download again It will continue from where it left off, but at regular (fast) download speed.

Re: Gemma3 – The current strongest model that fits on a single GPU

#25

Post by antirez »

sigmoid10 wrote: post #2

These bar charts are getting more disingenuous every day. This one makes it seem like Gemma3 ranks as nr. 2 on the arena just behind the full DeepSeek R1. But they just cut out everything that ranks higher. In reality, R1 currently ranks as nr. 6 in terms of Elo. It's still impressive for such a small model to compete with much bigger models, but at this point you can't trust any publication by anyone who has any ski…

The most disturbing thing is that in the chart it ranks higher than V3. Test a few prompts against DeepSeek V3 and Gemma 3. They are like at two totally different levels, one is a SOTA model, one is a small LLM that can be useful for certain vertical tasks perhaps.

Re: Gemma3 – The current strongest model that fits on a single GPU

#26

Post by antirez »

toinewx wrote: post #24
antirez wrote: post #22

After reading the technical report do the effort of downloading the model and run it against a few prompts. In 5 minutes you understand how broken LLM benchmarking is.

can you expand a bit?

The model performs very poorly in practice, while in the benchmark it is shown to be DeepSeek V3 level. It's not terrible but it's at another level compared to the models it is very close to (a bit better / a bit worse) in the benchmarks.

Re: Gemma3 – The current strongest model that fits on a single GPU

#28

Post by tomp »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Any below 7B you'd recommend?

IME Qwen2.5-3B-Instruct (or even 1.5B) have been quite remarkable, but I haven't done that heavy testing.

Post reply on HN