Gemini 2.5 Pro Preview

Post by obsolete_wagie » Tue, May 06, 2025, 8:00 PM UTC

mattlondon wrote: post #313

obsolete_wagie wrote: post #188

o3 is so far ahead of antrhopic and google, these models arent even worth using

The benchmarks (1) seem to suggest that o3 is in 3rd place after Gemini 2.5 pro preview and Gemini 2.5 pro exp (for text reasoning, o3 4th for webdev). o3 doesn't even appear on the openrouter leaderboards (2) suggesting is hardly used (if at all) by anyone using LLMs do actually do anything (such as coding) which makes one question if it is actually any good at all (otherwise if it was so great I'd expect to see hea…

Post by TuringNYC » Tue, May 06, 2025, 8:03 PM UTC

jjice wrote: post #326

jstummbillig wrote: post #314

Earlier quoted context omitted.

> no amount of prompting will get current models to approach abstraction and architecture the way a person does I find this sentiment increasingly worrisome. It's entirely clear that every last human will be beaten on code design in the upcoming years (I am not going to argue if it's 1 or 5 years away, who cares?) I wished people would just stop holding on to what amounts to nothing, and think and talk more about wha…

I'm confused by your comment. It seems like you didn't really provide a retort to the parent's comment about bad architecture and abstraction from LLMs. FWIW, I think you're probably right that we need to adapt, but there was no explanation as to _why_ you believe that that's the case.

Post by mdp2021 » Tue, May 06, 2025, 8:06 PM UTC

mattlondon wrote: post #308

kweingar wrote: post #64

Earlier quoted context omitted.

I expect my calculator to be 100% accurate 100% of the time. I have slightly more tolerance for other software having defects, but not much more.

AIs aren't intended to be used as calculators though? You could say that when I use my spanner/wrench to tighten a nut it works 100% of the time, but as soon as I try to use a screwdriver it's terrible and full of problems and it can't even reliably so something as trivially easy as tighten a nut, even though a screwdriver works the same way by using torque to tighten a fastener. Well that's because one tool is desig…

Post by shortcord » Tue, May 06, 2025, 8:07 PM UTC

paulirish wrote: post #250

> Gemini 2.5 Pro now ranks #1 on the WebDev Arena leaderboard It'd make sense to rename WebDev Arena to React/Tailwind Arena. Its system prompt requires [1] those technologies and the entire tool breaks when requesting vanilla JS or other frameworks. The second-order implications of models competing on this narrow definition of webdev are rather troublesome. [1] https://blog.lmarena.ai/blog/2025/webdev-arena/#:~:text…

Post by djhn » Tue, May 06, 2025, 8:13 PM UTC

dotancohen wrote: post #272

Jordan-117 wrote: post #247

Earlier quoted context omitted.

I recently needed to recommend some IAM permissions for an assistant on a hobby project; not complete access but just enough to do what was required. Was rusty with the console and didn't have direct access to it at the time, but figured it was a solid use case for LLMs since AWS is so ubiquitous and well-documented. I actually queried 4o, 3.7 Sonnet, and Gemini 2.5 for recommendations, stripped the list of duplicate…

AWS docs have (had) an embedded AI model that would do this perfectly. I suppose it had better training data, and the actual spec as a RAG.