Just wish iI could get 2.5 daily limit above 1000 requests easily. Driving me insane...
Gemini 3.1 Pro
401–410 of 951 posts
Re: Gemini 3.1 Pro
#402Earlier quoted context omitted.
I think we need to reevaluate what purpose these sorts of questions serve and why they're important in regards to judging intelligence. The model getting it correct or not at any given instance isn't the point, the point is if the model ever gets it wrong we can still assume that it still has some semblance of stochasticity in its output, given that a model is essentially static once it is released. Additionally, hey…
They're important but not as N=1. It's like cherry picking a single question from SimpleQA and going aha! It got it right! Meanwhile it's 8% lower score than some other model when evaluated on all questions.
Makes me wonder what people would consider better, a model that gets 92% of questions right 100% of the time, or a model that gets 95% of the questions right 90% of the time and 88% right the other 10%?
I think that's why benchmarking is so hard for me to fully get behind, even if we do it over say, 20 attempts and average it. For a given model, those 20 attempts could have had 5 incredible outcomes and 15 mediocre ones, whereas another model could have 20 consistently decent attempts and the average score would be generally the same.
We at least see variance in public benchmarks, but in the internal examples that's almost never the case.
Re: Gemini 3.1 Pro
#403Earlier quoted context omitted.
I had an interesting experience recently where I ran Opus 4.6 against a problem that o4-mini had previously convinced me wasn't tractable... and Opus 4.6 found me a great solution. https://github.com/simonw/sqlite-chronicle/issues/20 This inspired me to point the latest models at a bunch of my older projects, resulting in a flurry of fixes and unblocks.
I continue to get great value out of having claude and codex bound together in a loop: https://github.com/pjlsergeant/moarcode
They are one, the ring and the dark lord
Re: Gemini 3.1 Pro
#404Earlier quoted context omitted.
It's an excellent demonstration of the main issue I have with the Gemini family of models, they always go "above and beyond" to do a lot of stuff, even if I explicitly prompt against it. In this case, most of the SVG ends up consisting not just of a bike and a pelican, but clouds, a sun, a hat on the pelican and so much more. Exactly the same thing happens when you code, it's almost impossible to get Gemini to not do…
Would be really interesting to see an "Eager McBeaver" bench around this concept. When doing real work, a model's ability to stay within the bounds of a given task has almost become more important than its raw capabilities now that every frontier model is so dang good. Every one of these models is so great at propelling the ship forward, that I increasingly care more and more about which models are the easiest to ste…
being TOO steerable is another issue though.
Codex is very steerable to a fault, and will gladly "monkey paw" your requests to a fault.
Claude Opus will ignore your instructions and do what it thinks is "right" and just barrel forward.
Both are bad and papering over the actual issue which is these models don't really have the ability to actually selectively choose their behavior per issue (ie ask for followup where needed, ignore users where needed, follow instructions where needed). Behavior is largely global
Re: Gemini 3.1 Pro
#405Pretty great pelican: https://simonwillison.net/2026/Feb/19/gemini-31-pro/ - took over 5 minutes though, but I think that's because they're having performance teething problems on launch day.
How about STL files for 3d printing pelicans!
Harder: the bike must work
Hardest: the pelican must work
Re: Gemini 3.1 Pro
#406I hope this works better than 3.0 Pro I'm a former Googler and know some people near the team, so I mildly root for them to at least do well, but Gemini is consistently the most frustrating model I've used for development. It's stunningly good at reasoning, design, and generating the raw code, but it just falls over a lot when actually trying to get things done, especially compared to Claude Opus. Within VS Code Copi…
Gemini just doesn’t do even mildly well in agentic stuff and I don’t know why. OpenAI has mostly caught up with Claude in agentic stuff, but Google needs to be there and be there quickly
Because Search is not agentic.
Most of Gemini's users are Search converts doing extended-Search-like behaviors.
Agentic workflows are a VERY small percentage of all LLM usage at the moment. As that market becomes more important, Google will pour more resources into it.
Re: Gemini 3.1 Pro
#407Re: Gemini 3.1 Pro
#408Pretty great pelican: https://simonwillison.net/2026/Feb/19/gemini-31-pro/ - took over 5 minutes though, but I think that's because they're having performance teething problems on launch day.
What's crazy is you've influenced them to spend real effort ensuring their model is good at generating animated svgs of animals operating vehicles. The most absurd benchmaxxing. https://x.com/jeffdean/status/2024525132266688757?s=46&t=ZjF...
It if funny to think that Jeff Dean personally worked to optimize the pelican riding a bike benchmark.
Re: Gemini 3.1 Pro
#409Earlier quoted context omitted.
Gemini just doesn’t do even mildly well in agentic stuff and I don’t know why. OpenAI has mostly caught up with Claude in agentic stuff, but Google needs to be there and be there quickly
Can you explain what you mean by its bad at agentic stuff?
Accomplish the task I give to it without fighting me with it.
I think this is classic precision/recall issue: the model needs to stay on task, but also infer what user might want but not explicitly stated. Gemini seems particularly bad that recall, where it goes out of bounds
Re: Gemini 3.1 Pro
#410[dead]