Is "OpenAI" the only AI company that hasn't released any model weights?
Only if you give xAI credit for releasing grok-1, which is not a very useful model.
Gemma3 – The current strongest model that fits on a single GPU
41–50 of 148 posts
Re: Gemma3 – The current strongest model that fits on a single GPU
#42Is "OpenAI" the only AI company that hasn't released any model weights?
"AI company" makes this an unreasonable wide question but I'll assume you mean of the big players in this ecosystem. I miss later models from Grok and xai, which don't seem to care about sharing models either
Re: Gemma3 – The current strongest model that fits on a single GPU
#43I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…
TBH, I REALLY like the tiny models. Like smollm2. Also lobotomized LLMs ("abliterated") can be a lot of fun.
I think you mean un-lobotomize, and apparently it can be done without retraining? Wild!
Re: Gemma3 – The current strongest model that fits on a single GPU
#44I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…
Thanks for the overview.
> Qwen2.5-Coder-14B-Instruct - Good for simple coding tasks > OpenThinker-7B - Good and fast reasoning
Any chance you could be more specific, ie give an example of a concrete coding task or reasoning problem you used them for?
Re: Gemma3 – The current strongest model that fits on a single GPU
#45I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…
You should try Mistral Small 24b it’s been my daily companion for a while and have continued to impress me daily. I’ve heard good things about QwQ 32b that just came out too.
Re: Gemma3 – The current strongest model that fits on a single GPU
#46Earlier quoted context omitted.
They need to open source Sonnet 3.7. I know they won't, but a man can dream.
I'd wish people stop using "open sourcing" when speaking about models. Open sourcing is about being able to change and replicate builds, they make the models "freely available" but the recipe on how they are made is kept secret. It's akin to being able to download Windows shareware executables and calling that "open source" when nothing related to how the executables are build is available.
Well, yeah. They need to do that, too!
Re: Gemma3 – The current strongest model that fits on a single GPU
#47I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…
Do you mostly stick with smaller models? I’m pretty surprised at how good the smaller models can be at times now. A year ago they were nearly useless. I kind of like too the hallucinations are more obvious sometimes. Or at least it seems like they are.
I like the smaller models because they are faster. I even got a Llama 3 1B model running on TinkerBoard 2S and it was fun to play around with and not too slow. The smaller models are still good at summarizing and other basic tasks. For coding they start showing their limits but still work great for trying to figure out issues in small bits of code.
The real issue with local models is managing context. smaller models let you have a longer context without losing performance but bigger models are smarter but if you want to keep it fast I have to reduce the context length.
Also all of the models have their own "personalities" and they still manifest in the finetunes.
Re: Gemma3 – The current strongest model that fits on a single GPU
#48I found deepseek better for trivial tasks
Re: Gemma3 – The current strongest model that fits on a single GPU
#49I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…
Any below 7B you'd recommend? IME Qwen2.5-3B-Instruct (or even 1.5B) have been quite remarkable, but I haven't done that heavy testing.
Try;
- EXAONE-3.5-2.4B-Instruct - Llama-3.2-3B-Instruct-uncensored - qwq-lcot-3b-instruct - qwen2.5-3b-instruct
These have been very interesting tiny models, they can do text processing task and can handle story telling. The Llama-3.2 is way to sensitive to random stuff so get the uncensored or abliterated versions
Re: Gemma3 – The current strongest model that fits on a single GPU
#50I have tried a lot of local models. I have 656GB of them on my computer so I have experience with a diverse array of LLMs. Gemma has been nothing to write home about and has been disappointing every single time I have used it. Models that are worth writing home about are; EXAONE-3.5-7.8B-Instruct - It was excellent at taking podcast transcriptions and generating show notes and summaries. Rocinante-12B-v2i - Fun for s…
The Gemma 2 Instruct models are quite good (9 & 27B) for writing. The 27B is good at following instructions. I also like DeepSeek R1 Distill Llama 70B.
The Gemma 3 Instruct 4B model that was released today matches the output of the larger models for some of the stuff I am trying.
Recently, I compared 13 different online and local LLMs in a test where they tried to recreate Saki's "The Open Window" from a prompt.[1] Claude wins hands down IMO, but the other models are not bad.
[1] Variations on a Theme of Saki (https://gist.github.com/s-i-e-v-e/b4d696bfb08488aeb893cce3a4...)