Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma3 – The current strongest model that fits on a single GPU

ollama.com

Post reply on HN
Print view View original

91–100 of 148 posts

Re: Gemma3 – The current strongest model that fits on a single GPU

#91

Post by laborcontract »

michaelbuckbee wrote: post #88
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

What's the driving reason for local models? Cost? Censorship?

PII is the driving force for me. I like to have local models manage my browser tabs, reply to emails, and go through personal documents. I don't trust LLM providers not to retain my data.

Re: Gemma3 – The current strongest model that fits on a single GPU

#92

Post by tarruda »

My usual non-scientific benchmark is asking it to implement the game Tetris in python, and then iterating with the LLM to fix/tweak it.

My prompt to Gemma 27b (q4) on open webui + ollama: "Can you create the game tetris in python?"

It immediately starts writing code. After the code is finished, I noticed something very strange, it starts a paragraph like this:

" Key improvements and explanations:

     Clearer Code Structure:  The code is now organized into a Tetris class, making it much more maintainable and readable.  This is essential for any non-trivial game.
"

Followed by a bunch of fixes/improvements, as if this was not the first iteration of the script.

I also notice a very obvious error: In the `if __name__ == '__main__':` block, it tries to instantiate a `Tetris` class, when the name of the class it created was "TetrisGame".

Nevertheless, I try to run it and paste the `NameError: name 'Tetris' is not defined` error along with stack trace specifying the line. Gemma then gives me this response:

"The error message "NameError: name 'Tetris' is not defined" means that the Python interpreter cannot find a class or function named Tetris. This usually happens when:"

Then continues with a generic explanation with how to fix this error in arbitrary programs. It seems like it completely ignored the code it just wrote.

Re: Gemma3 – The current strongest model that fits on a single GPU

#94

Post by pzo »

anon373839 wrote: post #63
antirez wrote: post #26

Earlier quoted context omitted.

The model performs very poorly in practice, while in the benchmark it is shown to be DeepSeek V3 level. It's not terrible but it's at another level compared to the models it is very close to (a bit better / a bit worse) in the benchmarks.

I’d recommend trying it on Google AI Studio (aistudio.google.com). I am getting exceptional results on a handful of novel problems that require deep domain knowledge and structured reasoning. I’m not able to replicate this performance with Ollama, so I suspect something is a bit off.

Maybe model is sensitive to quantization, by default ollama quantize it significantly.

Re: Gemma3 – The current strongest model that fits on a single GPU

#95

Post by miroljub »

rpastuszak wrote: post #64
miroljub wrote: post #60

Earlier quoted context omitted.

Qwen2.5-Coder:32B is the best open source coding model. I use it daily, and I don't notice that it lags much behind Claude 3.5. I would be actually happy to see R1 distilled version, it may make it perform better with the less resource usage.

Thanks! Do you use it with Aider/terminal/a web GUI?

I use it with Emacs ellama, Continue.dev plugin and as a Web Chat.

Re: Gemma3 – The current strongest model that fits on a single GPU

#96

Post by panki27 »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

I've had really good results with Qwen2.5-7b-Instruct.

Do you have any recommendations for a "general AI assistant" model, not focused on a specific task, but more a jack-of-all-trades?

Re: Gemma3 – The current strongest model that fits on a single GPU

#97

Post by whiplash451 »

archerx wrote: post #56
aravindputrevu wrote: post #52

I'm curious. Is there any value to do these OSS models? Suddenly after reasoning models, it looks like OSS models have lost their charm

Thee are a lot of open source reasoning models. The true value to local models is privacy and the ability to have the models be uncensored.

Uncensored at inference time does not imply uncensored at training time (not a specific comment about Gemma)

Re: Gemma3 – The current strongest model that fits on a single GPU

#98

Post by tobacco-sausage »

CoryLR wrote: post #79
hnfong wrote: post #54

Earlier quoted context omitted.

If you actually looked at gemma-3 you’ll see that it does support system prompts. I’ve never seen a case where putting the system prompt in the user prompt would lead to significantly different outcomes though. Would like to see some examples. (edit: my bad. i stand corrected. it seems the code just prepends the system prompts to the first user prompt.)

This has been my experience as well. I don’t use system prompts anymore. Curious if there’s a good reason to start.

Direct prompt injection attacks can be somewhat mitigated by using system prompts (though not completely) if that helps.

Re: Gemma3 – The current strongest model that fits on a single GPU

#99

Post by flir »

karma_fountain wrote: post #78
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Ah, OpenThinker-7B. A diverse variety of LLM from the OpenThoughts team. Light and airy, suitable for everyday usage and not too heavy on the CPU. A new world LLM for the discerning user.

I find New World LLMs kinda... well, they don't have the terroir, ya know?

Re: Gemma3 – The current strongest model that fits on a single GPU

#100

Post by whiplash451 »

lelag wrote: post #67
archerx wrote: post #56

Earlier quoted context omitted.

Thee are a lot of open source reasoning models. The true value to local models is privacy and the ability to have the models be uncensored.

OSS model do not have to be local models, and it's not just about privacy, imo. DeepSeek R1 hosting is out of reach for most, but it being open is a game changer if you are a building a business that needs the SoTA capabilities of such a large model, not because you will necessarily host it yourself, but because you can't be locked out of using it. If you build your business on top of OpenAI, and they decide they don…

You can absolutely be locked out effectively if they stop releasing upgrades while the other providers move forward.
Post reply on HN