321–330 of 507 posts
Earlier quoted context omitted.
I am unable to shake that the Chinese models all perform awfully on the private arc-agi 2 tests.
But is arc-agi really that useful though? Nowadays it seems to me that it's just another benchmark that needs to be specifically trained for. Maybe the Chinese models just didn't focus on it as much.
I gave the same prompt (a small rust project that's not easy, but not overly sophisticated) to both Gemma-4 26b and Qwen 3.5 27b via OpenCode. Qwen 3.5 ran for a bit over an hour before I killed it, Gemma 4 ran for about 20 minutes before it gave up. Lots of failed tool calls. I asked codex to write a summary about both code bases. "Dev 1" Qwen 3.5 "Dev 2" Gemma 4 Dev 1 is the stronger engineer overall. They showed b…
There are issues with the chat template right now[0], so tool calling does not work reliably[1]. Every time people try to rush to judge open models on launch day... it never goes well. There are ~always bugs on launch day. [0]: https://github.com/ggml-org/llama.cpp/pull/21326 [1]: https://github.com/ggml-org/llama.cpp/issues/21316
Thinking / reasoning + multimodal + tool calling. We made some quants at https://huggingface.co/collections/unsloth/gemma-4 for folks to run them - they work really well! Guide for those interested: https://unsloth.ai/docs/models/gemma-4 Also note to use temperature = 1.0, top_p = 0.95, top_k = 64 and the EOS is " ". " thought\n" is also used for the thinking trace!
Earlier quoted context omitted.
Here's Gemma's. Qwen's is a bit too long to post here. Input format: ISO 8601 string (2026-04-01T16:00:00Z). Target format: Unix timestamp (seconds since Jan 1, 1970). Year: 2026 Month: 04 (April) Day: 01 Time: 16:00:00 Timezone: Z (UTC/Zulu) Option A: Manual calculation (too error-prone). Option B: Use a programming language (Python/JavaScript). Option C: Knowledge of specific dates. Let's use Python logic internall…
I find it interesting that it decided to write a script despite not having access to tools, and is apparently aware of this lack of access since it then proceeds to do the computation manually. It's impressive it got as close as it did with estimates (and that it can actually do basic math now). Yet then it goes "let's do a precise check using tools" and just blatantly makes the whole thing up. Comedic but also worri…
The difference between Gemma and Qwen here is that Qwen followed a much more detailed process - it consider leap years and seconds in its calculations (where Gemma used estimates like "roughly x years")
Thinking / reasoning + multimodal + tool calling. We made some quants at https://huggingface.co/collections/unsloth/gemma-4 for folks to run them - they work really well! Guide for those interested: https://unsloth.ai/docs/models/gemma-4 Also note to use temperature = 1.0, top_p = 0.95, top_k = 64 and the EOS is " ". " thought\n" is also used for the thinking trace!
Noob question. Why I would use this version over the original model?
Earlier quoted context omitted.
Doing great on public datasets and underperforming on private benchmarks is not a good look.
Is it though? Do we still have the expectation that LLMs will eventually be able to solve problems they haven't seen before? Or do we just want the most accurate auto complete at the cheapest price at this point?
Earlier quoted context omitted.
To clarify, the parent here didn't actually give the model a way to run the commands. The model just wrote the script/command and then, being unable to run anything, just mentally calculated what the result would probably be (and got it wrong). Yes the answer was wrong, but so was the setup (the model should have had access to a command runner tool).
Yes, you are right that for a model that wants to use tools, the environment was wrong. I didn't do that on purpose. I was simply interested in seeing what the answer to my question would be. The fact Gemma 4 wanted to use tools was a bit of a surprise to me - the Qwen model also can use tools, but it opted not to. I think it is interesting to see, that when forced to derive the value on its own, Gemma gets it wrong…
Earlier quoted context omitted.
Its hard to say because Pixel comes prepacked with a lot of models, not just ones that that are text output models. With the caveat that I'm not on the pixel team and I'm not building _all_ the models that are on google's devices, its evident there are many models that support the Android experience. For example the one mentioned here https://store.google.com/us/magazine/magic-editor?hl=en-US&p...
Yes of course, but I imagine there's only one main LLM on the device. Otherwise it's a waste of space to have multiple multi-gigabyte models that you then have to load into memory.