This has also been my biggest gripe with Gemini 2.5 Pro. While it is fantastic at one-shotting major new features, when wanting to make smaller iterative changes, it always does big refactors at the same time. I haven't found a way to change that behavior through changes in my prompts. Claude 3.7 Sonnet is much more restrained and does smaller changes.
Can't you just commit the relevant parts? The git index is made for this sort of thing.
It's not always trivial to find the relevant 5 line change in a diff of 200 lines...
My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…
Use few-shot learning. Build a simple prompt with basic examples of how to use the API and it will do significantly better.
LLMs just guess, so you have to give it a cheatsheet to help it guess closer to what you want.
It's already much improved on the early days. But I wonder when we'll be happy? Do we expect colleagues friends and family to be 100% laser-accurate 100% of the time? I'd wager we don't. Should we expect that from an artificial intelligence too?
If colleagues lie with the certainty that LLMs do, they would get fired for incompetence.
>Best-in-class frontend web development It really is wild to have seen this happen over the last year. The days of traditional "design-to-code" FE work are completely over. I haven't written a line of HTML/CSS in months. If you are still doing this stuff by hand, you need to adapt fast. In conjunction with an agentic coding IDE and a few MCP tools, weeks worth of UI work are now done in hours to a higher level of qua…
> are now done in hours to a higher level of quality However, I feel that there is a big difference between the models. In my tests, using Cursor, Clause 3.7 Sonnet has a much more refined "aesthetic sense" than other models. Many times I ask "make it more beautiful" and it manages to improve, where other models just can't understand it.
I've noticed the same, but I wonder if this new Gemini checkpoint is better at it now.
I use Gemini for almost everything. But their model card[1] only compares to o3-mini! In known benchmarks o3 is still ahead: +------------------------------+---------+--------------+ | Benchmark | o3 | Gemini 2.5 | | | | Pro | +------------------------------+---------+--------------+ | ARC-AGI (High Compute) | 87.5% | — | | GPQA Diamond (Science) | 87.7% | 84.0% | | AIME 2024 (Math) | 96.7% | 92.0% | | SWE-bench Veri…
The text in the model card says the results are from March (including the Gemini 2.5 Pro results), and o3 wasn't released yet.
Is this maybe not the updated card, even though the blog post claims there is one? Sure, the timestamp is in late April, but I seem to remember that the first model card for 2.5 Pro was only released in the last couple of weeks.
My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…
I recently needed to recommend some IAM permissions for an assistant on a hobby project; not complete access but just enough to do what was required. Was rusty with the console and didn't have direct access to it at the time, but figured it was a solid use case for LLMs since AWS is so ubiquitous and well-documented. I actually queried 4o, 3.7 Sonnet, and Gemini 2.5 for recommendations, stripped the list of duplicates, then passed the result to Gemini to vet and format as JSON. The result was perfectly formatted... and still contained a bunch of non-existent permissions. My first time being burned by a hallucination IRL, but just goes to show that even the latest models working in concert on a very well-defined problem space can screw up.
My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…
Use few-shot learning. Build a simple prompt with basic examples of how to use the API and it will do significantly better. LLMs just guess, so you have to give it a cheatsheet to help it guess closer to what you want.
My frustration with using these models for programming in the past has largely been around their tendency to hallucinate APIs that simply don't exist. The Gemini 2.5 models, both pro and flash, seem significantly less susceptible to this than any other model I've tried. There are still significant limitations, no amount of prompting will get current models to approach abstraction and architecture the way a person doe…
Use few-shot learning. Build a simple prompt with basic examples of how to use the API and it will do significantly better. LLMs just guess, so you have to give it a cheatsheet to help it guess closer to what you want.
At this point the time it takes to teach the model might be more than you save from using it for interacting with that API.
> Gemini 2.5 Pro now ranks #1 on the WebDev Arena leaderboard
It'd make sense to rename WebDev Arena to React/Tailwind Arena. Its system prompt requires [1] those technologies and the entire tool breaks when requesting vanilla JS or other frameworks. The second-order implications of models competing on this narrow definition of webdev are rather troublesome.