We didnt get deepseek v4, but gemma 4. Cant complain.
Google releases Gemma 4 open models
201–210 of 507 posts
Re: Google releases Gemma 4 open models
#202Comparison of Gemma 4 vs. Qwen 3.5 benchmarks, consolidated from their respective Hugging Face model cards: | Model | MMLUP | GPQA | LCB | ELO | TAU2 | MMMLU | HLE-n | HLE-t | |----------------|-------|-------|-------|------|-------|-------|-------|-------| | G4 31B | 85.2% | 84.3% | 80.0% | 2150 | 76.9% | 88.4% | 19.5% | 26.5% | | G4 26B A4B | 82.6% | 82.3% | 77.1% | 1718 | 68.2% | 86.3% | 8.7% | 17.2% | | G4 E4B |…
Reversing the X and Y axis, adding in a few other random models, and dropping all the small Qwens makes this worse than useless as a Qwen 3.5 comparison, it’s actively misleading. If you’re using AI, please don’t rush to copy paste output :/ EDIT: Lordy, the small models are a shadow of Qwen's smalls. See https://huggingface.co/Qwen/Qwen3.5-4B versus https://www.reddit.com/r/LocalLLaMA/comments/1salgre/gemma_4...
I transposed the table so that it's readable on mobile devices.
I should have mentioned that the Qwen 3.5 benchmarks were from the Qwen3.5-122B-A10B model card (which includes GPT-5-mini and GPT-OSS-120B); apologies for not including the smaller Qwen 3.5 models.
Re: Google releases Gemma 4 open models
#203Earlier quoted context omitted.
Thank you for your work. You have an answer on your page regarding "Should I pick 26B-A4B or 31B?", but can you please clarify if, assuming 24GB vRAM, I should pick a full precision smaller model or 4 bit larger model?
Thank you! I presume 24B is somewhat faster since it's only 4B activated - 31B is quite a large dense model so more accurate!
This is one of the more confusing aspects of experimenting with local models as a noob. Given my GPU, which model should I use, which quantization of that model should I pick (unsloth tends to offer over a dozen!) and what context size should I use? Overestimate any of these, and the model just won't load and you have to trial-and-error your way to finding a good combination. The red/yellow/green indicators on huggingface.co are kind of nice, but you only know for sure when you try to load the model and allocate context.
Re: Google releases Gemma 4 open models
#204Earlier quoted context omitted.
Expert streaming is something that has to be implemented by the inference engine/library, the model architecture itself has very little to do with it. It's a great idea (for local inference; it uses too much power at scale), but making it work really well is actually not that easy. (I've mentioned this before but AIUI it would require some new feature definitions in GGUF, to allow for coalescing model data about any…
I’ve been doing some low-key testing on smaller models, and it looks to me like it’s possible to train an MOE model with characteristics that are helpful for streaming… For instance, you could add a loss function to penalize expert swapping both in a single forward, pass and across multiple forward passes. So I believe there is a place for thinking about this on the model training side.
Penalizing expert swaps doesn't seem like it would help much, because experts vary by layer and are picked layer-wise. There's no guarantee that expert X in layer Y that was used for the previous token will still be available for this token's load from layer Y. The optimum would vary depending on how much memory you have at any given moment, and such. It's not obviously worth optimizing for.
Re: Google releases Gemma 4 open models
#205I gave the same prompt (a small rust project that's not easy, but not overly sophisticated) to both Gemma-4 26b and Qwen 3.5 27b via OpenCode. Qwen 3.5 ran for a bit over an hour before I killed it, Gemma 4 ran for about 20 minutes before it gave up. Lots of failed tool calls. I asked codex to write a summary about both code bases. "Dev 1" Qwen 3.5 "Dev 2" Gemma 4 Dev 1 is the stronger engineer overall. They showed b…
There are issues with the chat template right now[0], so tool calling does not work reliably[1].
Every time people try to rush to judge open models on launch day... it never goes well. There are ~always bugs on launch day.
Re: Google releases Gemma 4 open models
#206Earlier quoted context omitted.
But is arc-agi really that useful though? Nowadays it seems to me that it's just another benchmark that needs to be specifically trained for. Maybe the Chinese models just didn't focus on it as much.
Doing great on public datasets and underperforming on private benchmarks is not a good look.
Is it though? Do we still have the expectation that LLMs will eventually be able to solve problems they haven't seen before? Or do we just want the most accurate auto complete at the cheapest price at this point?
Re: Google releases Gemma 4 open models
#207So what's the business strategy here?
Google is the only USA based frontier lab releasing open models. I know they aren't doing it out of the goodness of their hearts.
Re: Google releases Gemma 4 open models
#208Earlier quoted context omitted.
Does spam filtering really need a better model? My impression is that the whole game is based on having the best and freshest user-contributed labels.
[flagged]
In my experience the contents of the message are all but totally irrelevant to the classification, and it is the behavior of the mailing peer that gives all the relevant features.
Re: Google releases Gemma 4 open models
#209Earlier quoted context omitted.
Reversing the X and Y axis, adding in a few other random models, and dropping all the small Qwens makes this worse than useless as a Qwen 3.5 comparison, it’s actively misleading. If you’re using AI, please don’t rush to copy paste output :/ EDIT: Lordy, the small models are a shadow of Qwen's smalls. See https://huggingface.co/Qwen/Qwen3.5-4B versus https://www.reddit.com/r/LocalLLaMA/comments/1salgre/gemma_4...
I transposed the table so that it's readable on mobile devices. I should have mentioned that the Qwen 3.5 benchmarks were from the Qwen3.5-122B-A10B model card (which includes GPT-5-mini and GPT-OSS-120B); apologies for not including the smaller Qwen 3.5 models.
It’s not readable on a phone either. Text wraps. unless you’re testing on foldable?
Re: Google releases Gemma 4 open models
#210Thinking / reasoning + multimodal + tool calling. We made some quants at https://huggingface.co/collections/unsloth/gemma-4 for folks to run them - they work really well! Guide for those interested: https://unsloth.ai/docs/models/gemma-4 Also note to use temperature = 1.0, top_p = 0.95, top_k = 64 and the EOS is " ". " thought\n" is also used for the thinking trace!
Hey, I tried to use Unsloth to run Gemma 4 locally but got stuck during the setup on Windows 11. At some point it asked me to create a password, and right after that it threw an error. Here’s a screenshot: https://imgur.com/a/sCMmqht This happened after running the PowerShell setup, where it installed several things like NVIDIA components, VS Code, and Python. At the end, PowerShell tell me to open a http://localhost…
Apologies we just fixed it!! If you try again from source ie
irm https://unsloth.ai/install.ps1 | iex
it should work hopefully. If not - please at us on Discord and we'll help you!
The Network error is a bummer - we'll check.
And yes we're working on a .exe!!