Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma3 – The current strongest model that fits on a single GPU

ollama.com

Post reply on HN
Print view View original

81–90 of 148 posts

Re: Gemma3 – The current strongest model that fits on a single GPU

#81

Post by m00dy »

pduggishetti wrote: post #15
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Recently phi4 has been very good too!

sshht, don't make it a public debate :P)

Re: Gemma3 – The current strongest model that fits on a single GPU

#82

Post by smahs »

sieve wrote: post #50
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

The Gemma 2 Instruct models are quite good (9 & 27B) for writing. The 27B is good at following instructions. I also like DeepSeek R1 Distill Llama 70B. The Gemma 3 Instruct 4B model that was released today matches the output of the larger models for some of the stuff I am trying. Recently, I compared 13 different online and local LLMs in a test where they tried to recreate Saki's "The Open Window" from a prompt.[1] C…

[dead]

Re: Gemma3 – The current strongest model that fits on a single GPU

#83

Post by sebastiansm »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Anyone can recommend a small model specific for translation? english to spanish mostly.

Re: Gemma3 – The current strongest model that fits on a single GPU

#84

Post by moffkalast »

CoryLR wrote: post #79
hnfong wrote: post #54

Earlier quoted context omitted.

If you actually looked at gemma-3 you’ll see that it does support system prompts. I’ve never seen a case where putting the system prompt in the user prompt would lead to significantly different outcomes though. Would like to see some examples. (edit: my bad. i stand corrected. it seems the code just prepends the system prompts to the first user prompt.)

This has been my experience as well. I don’t use system prompts anymore. Curious if there’s a good reason to start.

It matters as a common standard for model integration. System messages aren't removed when regular context messages start to get cut, the model pays more attention to tools and other directives defined there. Like situational context, language it should use, type of responses, etc. OAI/Anthropic give their models system prompts a mile long to tune their behaviour to a T.

There's also the ideological camp with Hartford and Nous and the rest where models are supposed to be trained as generally as possible, with the system prompt being strictly followed to adapt it to specific use cases.

Re: Gemma3 – The current strongest model that fits on a single GPU

#85

Post by dudefeliciano »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

what hardware are you using those on? Is it still prohibitively expensive to self-host a model that gives decent outputs (sorry my last experience has been underwhelming with llama a while back)

Re: Gemma3 – The current strongest model that fits on a single GPU

#86

Post by usef- »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

To clarify, are you basing this comment on experience with previous Gemma releases, or the one from today?

Re: Gemma3 – The current strongest model that fits on a single GPU

#87

Post by danielhanchen »

I wrote a mini guide on running Gemma 3 at https://docs.unsloth.ai/basics/tutorial-how-to-run-gemma-3-e...!

The recommended settings according to the Gemma team are:

temperature = 0.95

top_p = 0.95

top_k = 64

Also beware of double BOS tokens! You can run my uploaded GGUFs with the recommended chat template and settings via ollama run hf.co/unsloth/gemma-3-27b-it-GGUF:Q4_K_M

Re: Gemma3 – The current strongest model that fits on a single GPU

#88

Post by michaelbuckbee »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

What's the driving reason for local models? Cost? Censorship?

Re: Gemma3 – The current strongest model that fits on a single GPU

#89

Post by dannyw »

michaelbuckbee wrote: post #88
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

What's the driving reason for local models? Cost? Censorship?

Privacy is another big reason. I like to store my files locally with a backup, not on Dropbox or whatever.

Re: Gemma3 – The current strongest model that fits on a single GPU

#90

Post by pzo »

sebastiansm wrote: post #83
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Anyone can recommend a small model specific for translation? english to spanish mostly.

Depends what you mean small 4B? 7B? You can try qwen2.5 3B or 7B though 3B version is on no commercial license. Phi4-mini also should be good. Tested only on polish/english pairs should be good for spanish too. Smaller models like 1.5B were kind of useless for me.
Post reply on HN