Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemma3 – The current strongest model that fits on a single GPU

ollama.com

Post reply on HN
Print view View original

11–20 of 148 posts

Re: Gemma3 – The current strongest model that fits on a single GPU

#11

Post by mythz »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Concur with Gemma2 being underwhelming, I dismissed it pretty quickly but gemma3:27b is looking pretty good atm.

BTW mistral-small:24b is also worth mentioning (IMO best local model) and phi4:14b is also pretty strong for its size.

mistral-small was my previous local goto model, testing now to see if gemma3 can replace it.

Re: Gemma3 – The current strongest model that fits on a single GPU

#12

Post by wtcactus »

The claim of “strongest” (what does that even mean?) seems moot. I don’t think a multimodal model is the way to go to use on single, home, GPUs.

I would much rather have specific tailored models to use in different scenarios, that could be loaded into the GPU when needed. It’s a waste of parameters to have half of the VRAM loaded with parts of the model targeting image generation when all I want to do is write code.

Re: Gemma3 – The current strongest model that fits on a single GPU

#13

Post by Patrick_Devine »

mythz wrote: post #3

Not sure if anyone else experiences this, but ollama downloads starts off strong but the last few MBs take forever. Finally just finished downloading (gemma3:27b). Requires the latest version of Ollama to use, but now working, getting about 21 tok/s on my local 2x A4000. From my few test prompts looks like a quality model, going to run more tests to compare against mistral-small:24b to see if it's going to become my…

There are some fixes coming to uniformly speed up pulls. We've been testing that out but there are a lot of moving pieces with the new engine so it's not here quite yet.

Re: Gemma3 – The current strongest model that fits on a single GPU

#14

Post by InsideOutSanta »

mythz wrote: post #11
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Concur with Gemma2 being underwhelming, I dismissed it pretty quickly but gemma3:27b is looking pretty good atm. BTW mistral-small:24b is also worth mentioning (IMO best local model) and phi4:14b is also pretty strong for its size. mistral-small was my previous local goto model, testing now to see if gemma3 can replace it.

One more vote for Mistral for local models. The 7B model is extremely fast and still good enough for many prompts.

Re: Gemma3 – The current strongest model that fits on a single GPU

#15

Post by pduggishetti »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

Recently phi4 has been very good too!

Re: Gemma3 – The current strongest model that fits on a single GPU

#16

Post by anon373839 »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

From the limited testing I've done, Gemma 3 27B appears to be an incredibly strong model. But I'm not seeing the same performance in Ollama as I'm seeing on aistudio.google.com. So, I'd recommend trying it from the source before you draw any conclusions.

One of the downsides of open models is that there are a gazillion little parameters at inference time (sampling strategy, prompt template, etc.) that can easily impair a model's performance. It takes some time for the community to iron out the wrinkles.

Re: Gemma3 – The current strongest model that fits on a single GPU

#17

Post by [deleted] »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

[deleted]

Re: Gemma3 – The current strongest model that fits on a single GPU

#18

Post by DeepSeaTortoise »

archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

TBH, I REALLY like the tiny models. Like smollm2.

Also lobotomized LLMs ("abliterated") can be a lot of fun.

Re: Gemma3 – The current strongest model that fits on a single GPU

#19

Post by moffkalast »

anon373839 wrote: post #16
archerx wrote: post #6

I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…

From the limited testing I've done, Gemma 3 27B appears to be an incredibly strong model. But I'm not seeing the same performance in Ollama as I'm seeing on aistudio.google.com. So, I'd recommend trying it from the source before you draw any conclusions. One of the downsides of open models is that there are a gazillion little parameters at inference time (sampling strategy, prompt template, etc.) that can easily impa…

At the end of the day it doesn't matter how good it its, it has no system prompt which means no steerability, a sliding window for incredibly slow inference compared to similar sized models because it's too niche and most inference systems have high overhead implementations of it, and Google's psychotic instruct tuning that made Gemma 2 an inconsistent and unreliable glass cannon.

I mean hell, even Mistral added system prompts in their last release, Google are the only ones that don't seem to bother with it by now.

Post reply on HN