How does this compare with qwq 32B?
Gemma3 – The current strongest model that fits on a single GPU
71–80 of 148 posts
Re: Gemma3 – The current strongest model that fits on a single GPU
#72Discrete GPUs are finished for AI. They've had years to provide the needed memory but can't/won't. The future of local LLMs is APUs such as Apple M series and AMD Strix Halo. Within 12 months everyone will have relegated discrete GPUs to the AI dustbin and be running 128GB to 512GB of delicious local RAM with vastly more RAM than any discrete GPU could dream of.
That seems a tad dramatic. GPU's were widespread because of gaming, not AI. That the overlapping market would somehow just all magically have >3,000$ _and_ decide to switch to a non-standard, non-CUDA hardware solution in just 12 months is absurd.
Re: Gemma3 – The current strongest model that fits on a single GPU
#73After reading the technical report do the effort of downloading the model and run it against a few prompts. In 5 minutes you understand how broken LLM benchmarking is.
Aren't there any "blind" benchmarks?
Re: Gemma3 – The current strongest model that fits on a single GPU
#74Good job Google. It is kinda hilarious that 'open'AI seems to be the big player least likely to release any of their models.
lyingAI
Re: Gemma3 – The current strongest model that fits on a single GPU
#75The claim of “strongest” (what does that even mean?) seems moot. I don’t think a multimodal model is the way to go to use on single, home, GPUs. I would much rather have specific tailored models to use in different scenarios, that could be loaded into the GPU when needed. It’s a waste of parameters to have half of the VRAM loaded with parts of the model targeting image generation when all I want to do is write code.
Perhaps the model performs better (has higher intelligence) if it was trained on a more diverse set of topics (?)
Re: Gemma3 – The current strongest model that fits on a single GPU
#76PSA: DO NOT USE OLLAMA FOR TESTING.
Ollama silently (!!!) drops messages if the context window is exceeded (instead of, you know, just erroring? who in the world made this decision).
The workaround until now was to (not use ollama or) make sure to only send a single message. But now they seem to silently truncate single messages as well, instead of erroring! (this explains the sibling comment where a user could not reproduce the results locally).
Use LM Studio, llama.cpp, openrouter or anything else, but stay away from ollama!
Re: Gemma3 – The current strongest model that fits on a single GPU
#77Earlier quoted context omitted.
At the end of the day it doesn't matter how good it its, it has no system prompt which means no steerability, a sliding window for incredibly slow inference compared to similar sized models because it's too niche and most inference systems have high overhead implementations of it, and Google's psychotic instruct tuning that made Gemma 2 an inconsistent and unreliable glass cannon. I mean hell, even Mistral added syst…
If you actually looked at gemma-3 you’ll see that it does support system prompts. I’ve never seen a case where putting the system prompt in the user prompt would lead to significantly different outcomes though. Would like to see some examples. (edit: my bad. i stand corrected. it seems the code just prepends the system prompts to the first user prompt.)
I've read the Gemma 3 technical report, it doesn't mention anything about it in the format section. Did they forget to include that? Where did you find the source that claims otherwise?
Re: Gemma3 – The current strongest model that fits on a single GPU
#78I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…
Ah, OpenThinker-7B. A diverse variety of LLM from the OpenThoughts team. Light and airy, suitable for everyday usage and not too heavy on the CPU. A new world LLM for the discerning user.
Re: Gemma3 – The current strongest model that fits on a single GPU
#79Earlier quoted context omitted.
At the end of the day it doesn't matter how good it its, it has no system prompt which means no steerability, a sliding window for incredibly slow inference compared to similar sized models because it's too niche and most inference systems have high overhead implementations of it, and Google's psychotic instruct tuning that made Gemma 2 an inconsistent and unreliable glass cannon. I mean hell, even Mistral added syst…
If you actually looked at gemma-3 you’ll see that it does support system prompts. I’ve never seen a case where putting the system prompt in the user prompt would lead to significantly different outcomes though. Would like to see some examples. (edit: my bad. i stand corrected. it seems the code just prepends the system prompts to the first user prompt.)
This has been my experience as well. I don’t use system prompts anymore. Curious if there’s a good reason to start.
Re: Gemma3 – The current strongest model that fits on a single GPU
#80Discrete GPUs are finished for AI. They've had years to provide the needed memory but can't/won't. The future of local LLMs is APUs such as Apple M series and AMD Strix Halo. Within 12 months everyone will have relegated discrete GPUs to the AI dustbin and be running 128GB to 512GB of delicious local RAM with vastly more RAM than any discrete GPU could dream of.
FWIW GPUs still do not saturate PCIe lanes.