For Gemini: play around with the temperature: the default is terrible: we had much better results with (much) lower values.
Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
11–20 of 336 posts
Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
#12From my use case, the Gemini 2.5 is terrible. I have a complex Cython code in a single file (1500 lines) for a Sequence Labeling. Claude and o3 are very good in improving this code and following the commands. The Gemini always try to do unrelated changes. For example, I asked, separately, for small changes such as remove this unused function, or cache the arrays indexes. Every time it completely refactored the code a…
This reflects my experience 1:1... even telling 2.5 Pro to focus on the tasks given and ignore everything else leads to it changing unrelated code.
It's a frustrating experience because I believe at its core it is more capable than Sonnet 3.5/3.7
Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
#13Gemini is the only model which tells me when it's a good time to stop chatting because either it can't find a solution or because it dislikes my solution (when I actively want to neglect security).
And the context length is just amazing. When ChatGPT's context is full, it totally forgets what we were chatting about, as if it would start an entirely new chat.
Gemini lacks the tooling, there ChatGPT is far ahead, but at its core, Gemini feels like a better model.
Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
#14I've been coding with both non stop the last few days, gemini 2.5 pro is not even close. For complicated bug solving, o1 pro is still far ahead of both. Sonnet 3.7 is best overall
Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
#15From my use case, the Gemini 2.5 is terrible. I have a complex Cython code in a single file (1500 lines) for a Sequence Labeling. Claude and o3 are very good in improving this code and following the commands. The Gemini always try to do unrelated changes. For example, I asked, separately, for small changes such as remove this unused function, or cache the arrays indexes. Every time it completely refactored the code a…
Yup, gemini 2.5 is bad.
Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
#16From my use case, the Gemini 2.5 is terrible. I have a complex Cython code in a single file (1500 lines) for a Sequence Labeling. Claude and o3 are very good in improving this code and following the commands. The Gemini always try to do unrelated changes. For example, I asked, separately, for small changes such as remove this unused function, or cache the arrays indexes. Every time it completely refactored the code a…
How are you asking Gemini 2.5 to change existing code? With Claude 3.7, it's possible to use Claude Code, which gets "extremely fast but untrustworthy intern"-level results. Do you have a prefered setup to use Gemini 2.5 in a similar agentic mode, perhaps using a tool like Cursor or aider?
Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
#17TL;DR If you want to jump straight to the conclusion, I’d say go for Gemini 2.5 Pro, it’s better at coding, has one million in context window as compared to Claude’s 200k, and you can get it for free (a big plus). However, Claude’s 3.7 Sonnet is not that far behind. Though at this point there’s no point using it over Gemini 2.5 Pro.
> has one million in context window Is this effective context window or just the absolute limit? A lot of the models that claim to support very large context windows cannot actually successfully do the typical "needle in a haystack" test, but I'm guessing there are published results somewhere demonstrating Gemini 2.5 Pro can actually find the needle?
This is a good question. There's a big difference in being able to write coherent code and "needle in the haystack" questions. I've found that Claude is able to do the needle in the haystack questions just fine with a large context, but not so with coding. You have to work to keep the context low (around 15% to 20% in projects) to get coherent code that doesn't confabulate.
Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
#18I've been coding with both non stop the last few days, gemini 2.5 pro is not even close. For complicated bug solving, o1 pro is still far ahead of both. Sonnet 3.7 is best overall
I think O1 Pro Mode is so infrequently used by others (because of the price) so I've just started added "besides O1 Pro Mode, if you have access" in my head when someone says "This is the best available model for X".
It really is miles ahead of anything else so far, but also really pricey so makes sense some people try to find something close to it with much lower costs.
Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
#19I noticed a similar trends in selling on X. Put a claim, peg on some product A with good sales - Cursor, Claude, Gemini, etc. Then say, the best way to use A is with our best product, guide, being MCP or something else.
For some of these I see something like 15k followers on X, but then no LinkedIn page for example. Website is always a company you cannot contact and they do everything.
Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison
#20This is not a good comparison for real world coding tasks.
Based on my own experience and anectodes, it's worse than Claude 3.5 and 3.7 Sonnet for actual coding tasks on existing projects. It is very difficult to control the model behavior.
I will probably make a blog post on real world usage.