Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

composio.dev

Post reply on HN
Print view View original

21–30 of 336 posts

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#21

Post by llm_nerd »

diggan wrote: post #8
mraniki wrote: post #2

TL;DR If you want to jump straight to the conclusion, I’d say go for Gemini 2.5 Pro, it’s better at coding, has one million in context window as compared to Claude’s 200k, and you can get it for free (a big plus). However, Claude’s 3.7 Sonnet is not that far behind. Though at this point there’s no point using it over Gemini 2.5 Pro.

> has one million in context window Is this effective context window or just the absolute limit? A lot of the models that claim to support very large context windows cannot actually successfully do the typical "needle in a haystack" test, but I'm guessing there are published results somewhere demonstrating Gemini 2.5 Pro can actually find the needle?

Google has had almost perfect recall in the needle in the haystack test since 1.5[1], achieving close to 100% over the entire context window. I can't provide a link benchmarking 2.5 Pro in particular, but this has been a solved problem with Google models so I assume the same is true with their new model.

[1] https://cloud.google.com/blog/products/ai-machine-learning/t...

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#22

Post by bratao »

ekidd wrote: post #16
bratao wrote: post #5

From my use case, the Gemini 2.5 is terrible. I have a complex Cython code in a single file (1500 lines) for a Sequence Labeling. Claude and o3 are very good in improving this code and following the commands. The Gemini always try to do unrelated changes. For example, I asked, separately, for small changes such as remove this unused function, or cache the arrays indexes. Every time it completely refactored the code a…

How are you asking Gemini 2.5 to change existing code? With Claude 3.7, it's possible to use Claude Code, which gets "extremely fast but untrustworthy intern"-level results. Do you have a prefered setup to use Gemini 2.5 in a similar agentic mode, perhaps using a tool like Cursor or aider?

For all LLMs, I´m using a simple prompt with the complete code in triple quotes and the command at the end, asking to output the complete code of changed functions. Then I use Winmerge to compare the changes and apply. I feel more confident doing this than using Cursor.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#24

Post by redog »

bratao wrote: post #5

From my use case, the Gemini 2.5 is terrible. I have a complex Cython code in a single file (1500 lines) for a Sequence Labeling. Claude and o3 are very good in improving this code and following the commands. The Gemini always try to do unrelated changes. For example, I asked, separately, for small changes such as remove this unused function, or cache the arrays indexes. Every time it completely refactored the code a…

For me I had to upload the library's current documentation to it because it was using outdated references and changing everything that was working in the code to broken and not focusing on the parts I was trying to build upon.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#25

Post by Jowsey »

polycaster wrote: post #7
mraniki wrote: post #2

TL;DR If you want to jump straight to the conclusion, I’d say go for Gemini 2.5 Pro, it’s better at coding, has one million in context window as compared to Claude’s 200k, and you can get it for free (a big plus). However, Claude’s 3.7 Sonnet is not that far behind. Though at this point there’s no point using it over Gemini 2.5 Pro.

If there'd just be an alternative to claude code...

Isn't https://aider.chat similar?

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#26

Post by MITSardine »

mraniki wrote: post #2

TL;DR If you want to jump straight to the conclusion, I’d say go for Gemini 2.5 Pro, it’s better at coding, has one million in context window as compared to Claude’s 200k, and you can get it for free (a big plus). However, Claude’s 3.7 Sonnet is not that far behind. Though at this point there’s no point using it over Gemini 2.5 Pro.

What does this context window mean, is it the size of the prompt it can be made aware of?

In practice, can you use any of these models with existing code bases of, say, 50k LoC?

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#27

Post by Sol- »

Maybe I don't feel the AI FOMO strongly enough and obviously these performance comparisons can be interesting in their own right to keep track of AI progress, but ultimately it feels as long as you have a pro subscription of one of the leading providers (OpenAI, Anthropic or Google), you're fine.

Sure, your provider of choice might fall behind for a few months, but they'll just release a new version eventually and might come out on top again. Intelligence seems commodified enough already that I don't care as much whether I have the best or second best.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#29

Post by neal_ »

I was using gemini 2.5 pro yesterday and it does seem decent. I still think claude 3.5 is better at following instruction then the new 3.7 model which just goes ham messing stuff up. Really disappointed by Cursor and the Claude CLI tool, for me they create more problems then fix. I cant figure out how to use them on any of my projects with out them ruining the project and creating terrible tech debt. I really like the way gemini shows how much context window is left, i think every company should have this. To be honest i think there has been no major improvement beyond the original models which gained popularity first. Its just marginal improvements 10% better or something, and the free models like deepseek are actually better imo then anything openai has. I dont think the market can withstand the valuations of the big ai companies. They have no advantage, there models suck worse then free open source ones, and they charge money??? Where is the benefit to there product?? People originally said the models are the moat and methods are top secret, but turns out its pretty easy to reproduce these models, and its the application layer built on top of the models that is much more specific and has the real moat. People said the models would engulf these applications built ontop and just integrate natively.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#30

Post by ldjkfkdsjnv »

diggan wrote: post #18
ldjkfkdsjnv wrote: post #14

I've been coding with both non stop the last few days, gemini 2.5 pro is not even close. For complicated bug solving, o1 pro is still far ahead of both. Sonnet 3.7 is best overall

I think O1 Pro Mode is so infrequently used by others (because of the price) so I've just started added "besides O1 Pro Mode, if you have access" in my head when someone says "This is the best available model for X". It really is miles ahead of anything else so far, but also really pricey so makes sense some people try to find something close to it with much lower costs.

Yeah its not even close. In my mind, the 200$ a month could be 500 and I would still pay for it. There are many technical problems I have ran into, where I simply would not have solved the problem without it. I am building more complicated software than I ever have, and I have 10+ years of engineering experience in big tech
Post reply on HN