Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

composio.dev

Post reply on HN
Print view View original

221–230 of 336 posts

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#221

Post by anotherpaulg »

Gemini 2.5 Pro set a wide SOTA on the aider polyglot coding leaderboard [0]. It scored 73%, well ahead of the previous 65% SOTA from Sonnet 3.7.

I use LLMs to improve aider, which is >30k lines of python. So not a toy codebase, not greenfield.

I used Gemini 2.5 Pro for the majority of the work on the latest aider release [1]. This is the first release in a very long time which wasn't predominantly written using Sonnet.

The biggest challenge with Gemini right now is the very tight rate limits. Most of my Sonnet usage lately is just when I am waiting for Gemini’s rate limits to cool down.

[0] https://aider.chat/docs/leaderboards/

[1] https://aider.chat/docs/faq.html#what-llms-do-you-use-to-bui...

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#222

Post by dimitri-vs »

mountainriver wrote: post #73
neal_ wrote: post #29

I was using gemini 2.5 pro yesterday and it does seem decent. I still think claude 3.5 is better at following instruction then the new 3.7 model which just goes ham messing stuff up. Really disappointed by Cursor and the Claude CLI tool, for me they create more problems then fix. I cant figure out how to use them on any of my projects with out them ruining the project and creating terrible tech debt. I really like th…

My whole team feels like 3.7 is a letdown. It really struggles to follow instructions as others are mentioning. Makes me think they really just hacked the benchmarks on this one.

They definitely over-optimized it for agentic use - where the quality of the code doesn't matter as much as it's ability to run, even if just barely. When you view it from that perspective all that nested errors handling, excessive comments, 10 lines that can be done in 2, etc. start to make sense.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#223

Post by neal_ »

heed wrote: post #183
cjonas wrote: post #46

Earlier quoted context omitted.

My only experience is via cursor but I'd agree in that context 3.7 is worse than 3.5. 3.7 goes crazy trying to fix any little linter errors and often gets confused and will just hammer away, making things worse until I stop generation. I think if I let it continue it would probably proposed rm -rf and start over at some point :). Again, this could just have to do with the way cursor is prompting it.

believe it or not, i had cursor in yolo mode just for fun recently and 3.7 rm -rf'd my home folder :(

thats crazy! I haven't heard of yolo mode?? dont they like restrict access to the project? but i guess the terminal is unrestricted? lol i wonder what it was trying to do

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#224

Post by nathanasmith »

Extropy_ wrote: post #196

Why is Grok not in their benchmarks? I don't see comparisons to Grok in any recent announcements about models. In fact, I see practically no discussion of Grok on HN or anywhere except Twitter in general.

Is there an API for Grok yet? If not that could be the issue.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#225

Post by Philpax »

semi-extrinsic wrote: post #184
ramesh31 wrote: post #84

Earlier quoted context omitted.

You guys really need a Docker build. This dependency chain with submodules is a nightmare.

Alternative perspective: you kids with your Docker builds need to roll up your sleeves and learn how to actually compile a semi-complicated project if you expect to be able to contribute back to said project.

If your project is hard to build, that's your problem, not mine. I'll simply spend my time working on projects that respect it.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#226

Post by marlott »

breadwinner wrote: post #214
gnatolf wrote: post #204

Earlier quoted context omitted.

Any way you can back up that Copilot is a flop?

Lots of articles on it... and I am not even talking about competitors like Benioff [1]. I am talking about user complaints like this [2]. Users expect Copilot to be fully integrated, like Cursor is into VSCode. Instead what you get is barely better than typing into standalone AI chats like Claude.AI. [1] https://www.cio.com/article/3586887/marc-benioff-rails-again... [2] https://techcommunity.microsoft.com/discussion…

[dead]

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#227

Post by InTheArena »

The amazing bit about claude code is it's ability to read code, and fit into the existing code base. I tried visual studio code w/ roo, and it blew up my 50 daily request limit immediately. Any suggestions on better tooling for a claude code like experience with Gemeni 2.5 pro?

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#228

Post by genewitch »

Jcampuzano2 wrote: post #99
redog wrote: post #24

Earlier quoted context omitted.

For me I had to upload the library's current documentation to it because it was using outdated references and changing everything that was working in the code to broken and not focusing on the parts I was trying to build upon.

If you don't mind me asking how do you go about this? I hear people commonly mention doing this but I can't imagine people are manually adding every page of the docs for libraries or frameworks they're using since unfortunately most are not in one single tidy page easy to copy paste.

Have the AI write a quick script using bs4 or whatever to take the HTML dump and output json, then all the aider-likes can use that json as documentation. Or just the HTML, but that wastes context window.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#229

Post by genewitch »

throwup238 wrote: post #141
lionkor wrote: post #126

Earlier quoted context omitted.

If I give you hundreds millions of dollars for just making a clone of something that exists (an LLM) and hype the shit out of it, how far would you go?

I would change the world™ and make it a better place®.

Empowering everyone to bring their ideas to life

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#230

Post by phkahler »

amelius wrote: post #143
phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

FWIW, what I want most in Solvespace is a way to do chamfers and fillets. And a way to define parameters (not sure if that's already possible).

>> FWIW, what I want most in Solvespace is a way to do chamfers and fillets.

I've outlined a function for that and started to write the code. At a high level it's straight forward, but the details are complex. It'll probably be a year before it's done.

>> And a way to define parameters (not sure if that's already possible).

This is an active work in progress. A demo was made years ago, but it's buggy and incomplete. We've been working out the details on how to make it work. I hope to get the units issue dealt with this week. Then the relation constraints can be re-integrated on top - that's the feature where you can type arbitrary equations on the sketch using named parameters (variables). I'd like that to be done this year if not this summer.

Post reply on HN