Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma3 – The current strongest model that fits on a single GPU

ollama.com

Post reply on HN
Print view View original

111–120 of 148 posts

Re: Gemma3 – The current strongest model that fits on a single GPU

#111

Post by moffkalast »

staticman2 wrote: post #109
moffkalast wrote: post #77

Earlier quoted context omitted.

I've read the Gemma 3 technical report, it doesn't mention anything about it in the format section. Did they forget to include that? Where did you find the source that claims otherwise?

Google AI Studio offers no system prompt option for Gemma 3 which also suggests this model doesn't support it...

The official ggufs have this in them as an example, I'm even more confused now:

user

You are a helpful assistant

Hello

model

Hi there

user

How are you?

model

Re: Gemma3 – The current strongest model that fits on a single GPU

#112

Post by hnfong »

moffkalast wrote: post #111
staticman2 wrote: post #109

Earlier quoted context omitted.

Google AI Studio offers no system prompt option for Gemma 3 which also suggests this model doesn't support it...

The official ggufs have this in them as an example, I'm even more confused now: user You are a helpful assistant Hello model Hi there user How are you? model

I was confused by this too, but now I think they’re just suggesting to prepend the system prompt to the first user prompt.

I think the ggufs took the chat template defined in tokenizer_config.json , which basically does that.

Re: Gemma3 – The current strongest model that fits on a single GPU

#113

Post by singularity2001 »

How does it compare to OlympicCoder 7B [0] which allegedly beats Claude Sonnet 3.7 in the International Olympiad in Informatics [1] ?

[0] https://huggingface.co/open-r1/OlympicCoder-7B?local-app=vll...

[1] https://pbs.twimg.com/media/GlyjSTtXYAAR188?format=jpg&name=...

Re: Gemma3 – The current strongest model that fits on a single GPU

#114

Post by pcdoodle »

alekandreev wrote: post #51
antirez wrote: post #26

Earlier quoted context omitted.

The model performs very poorly in practice, while in the benchmark it is shown to be DeepSeek V3 level. It's not terrible but it's at another level compared to the models it is very close to (a bit better / a bit worse) in the benchmarks.

Hey, Gemma engineer here. Can you please share reports on the type of prompts and the implementation you used?

[flagged]

Re: Gemma3 – The current strongest model that fits on a single GPU

#115

Post by tarruda »

j_maffe wrote: post #106
toinewx wrote: post #66

Earlier quoted context omitted.

would you be able to run Sonnet 3.7 on a consumer computer though?

You can't with DeepSeek either but it has aided open source models significantly by being open source.

The new M3 Ultra Mac Studio (512GB version) seems to be capable of running DeepSeek R1 in Q4

Re: Gemma3 – The current strongest model that fits on a single GPU

#116

Post by whbrown »

tarruda wrote: post #92

My usual non-scientific benchmark is asking it to implement the game Tetris in python, and then iterating with the LLM to fix/tweak it. My prompt to Gemma 27b (q4) on open webui + ollama: "Can you create the game tetris in python?" It immediately starts writing code. After the code is finished, I noticed something very strange, it starts a paragraph like this: " Key improvements and explanations: Clearer Code Structu…

Those sound like the sort of issues which could be caused by your server silently truncating the middle of your prompts.

By default, Ollama uses a context window size of 2048 tokens.

Re: Gemma3 – The current strongest model that fits on a single GPU

#117

Post by sgt101 »

alekandreev wrote: post #51
antirez wrote: post #26

Earlier quoted context omitted.

The model performs very poorly in practice, while in the benchmark it is shown to be DeepSeek V3 level. It's not terrible but it's at another level compared to the models it is very close to (a bit better / a bit worse) in the benchmarks.

Hey, Gemma engineer here. Can you please share reports on the type of prompts and the implementation you used?

vibe testing, vibe model engineering...

Re: Gemma3 – The current strongest model that fits on a single GPU

#118

Post by archerx »

panki27 wrote: post #96
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

I've had really good results with Qwen2.5-7b-Instruct. Do you have any recommendations for a "general AI assistant" model, not focused on a specific task, but more a jack-of-all-trades?

If I could only use one model from now on it would either be the deepSeek R1 Qwen or Llama distill.

Re: Gemma3 – The current strongest model that fits on a single GPU

#119

Post by archerx »

sebastiansm wrote: post #83
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Anyone can recommend a small model specific for translation? english to spanish mostly.

I haven't done deep testing on it but Tower-Babel_Babel-9B should be what you are looking for.

Re: Gemma3 – The current strongest model that fits on a single GPU

#120

Post by xnx »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Let us know when you've evaluated Gemma 3. Just as with the switch between ChatGPT 3.5 and ChatGPT 4, old versions don't tell you much about the current version.
Post reply on HN