Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

composio.dev

Post reply on HN
Print view View original

111–120 of 336 posts

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#111

Post by SV_BubbleTime »

snickell wrote: post #96
phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

This is the smoothest tom sawyer move I've ever seen IRL, I wonder how many people are now grinding out your GTK4 port with our favorite LLM/system to see if it can. It'll be interesting to see if anyone gets something working with current-gen LLMs. UPDATE: naive (just fed it your description verbatim) cline + claude 3.7 was a total wipeout. It looked like it was making progress, then freaked out, deleted 3/4 of its…

Smooth? Nah.

Tom Sawyer? Yes.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#112

Post by pera »

gavinray wrote: post #69
phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

Convert the GTK 3 and GTK 4 API documentation into a single `.txt` file each. Upload one of your platform-specific C++ file's source, along with the doc `.txt` into your LLM of choice. Either ask it for a conversion function-by-function, or separate it some other way logically such that the output doesn't get truncated. Would be surprised if this didn't work, to be honest.

Do you really need to provide the docs? I would have imagined that those docs are included in their training sets. There is even a guide on how to migrate from GTK3 to GTK4, so this seems to be a low-hanging fruit job for an LLM iff they are okay for coding.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#113

Post by phkahler »

snickell wrote: post #96
phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

This is the smoothest tom sawyer move I've ever seen IRL, I wonder how many people are now grinding out your GTK4 port with our favorite LLM/system to see if it can. It'll be interesting to see if anyone gets something working with current-gen LLMs. UPDATE: naive (just fed it your description verbatim) cline + claude 3.7 was a total wipeout. It looked like it was making progress, then freaked out, deleted 3/4 of its…

>> This is the smoothest tom sawyer move I've ever seen IRL

That made me laugh. True, but not really the motivation. I honestly don't think LLMs can code significant real-world things yet and I'm not sure how else to prove that since they can code some interesting things. All the talk about putting programmers out of work has me calling BS but also thinking "show me". This task seems like a good combination of simple requirements, not much documentation, real world existing problem, non-trivial code size, limited scope.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#114

Post by lherron »

These one-shot prompts aren't at all how most engineers use these models for coding. In my experience so far, Gemini 2.5 Pro is great at generating code but not so great at instruction following or tool usage, which are key for any iterative coding tasks. Claude is still king for that reason.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#115

Post by travisgriggs »

cjonas wrote: post #46
neal_ wrote: post #29

I was using gemini 2.5 pro yesterday and it does seem decent. I still think claude 3.5 is better at following instruction then the new 3.7 model which just goes ham messing stuff up. Really disappointed by Cursor and the Claude CLI tool, for me they create more problems then fix. I cant figure out how to use them on any of my projects with out them ruining the project and creating terrible tech debt. I really like th…

My only experience is via cursor but I'd agree in that context 3.7 is worse than 3.5. 3.7 goes crazy trying to fix any little linter errors and often gets confused and will just hammer away, making things worse until I stop generation. I think if I let it continue it would probably proposed rm -rf and start over at some point :). Again, this could just have to do with the way cursor is prompting it.

So glad to see this!! I thought it was just me!

The latest updates, I’m often like “would you just hold the f#^^ on trigger?!? Take a chill pill already”

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#116

Post by otabdeveloper4 »

Workaccount2 wrote: post #62
amazingamazing wrote: post #23

In before people post contradictory anecdotes. It would be more helpful if people posted the prompt, and the entire context, or better yet the conversation, so we can all judge for ourselves.

This is also compounded by the fact that LLMs are not deterministic, every response is different for the same given prompt. And people tend to judge on one off experiences.

> LLMs are not deterministic

They can be. The cloud-hosted LLMs add a gratuitous randomization step to make the output seem more human. (In vein with the moronic idea of selling LLM's as sci-fi human-like assistants.)

But you don't have to add those randomizations. Nothing much is lost if you don't. (Output from my self-hosted LLM's is deterministic.)

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#117

Post by light_hue_1 »

qwertox wrote: post #85
criddell wrote: post #74

Earlier quoted context omitted.

I asked Claude this weekend what it could tell me about writing Paint.Net plugins and it responded that it didn't know much about that: > I'd be happy to help you with information about writing plugins for Paint.NET. This is a topic I don't have extensive details on in my training, so I'd like to search for more current information. Would you like me to look up how to create plugins for Paint.NET?

I mean responses like this one: I understand the desire for a simple or unconventional solution, however there are problems with those solutions. There is likely no further explanation that will be provided. It is best that you perform testing on your own. Good luck, and there will be no more assistance offered. You are likely on your own. This was about a SOCKS proxy which was leaking when the OpenVPN provider was d…

You like that?

This junk is why I don't use Gemini. This isn't a feature. It's a fatal bug.

It decides how things should go, if its way is right, and if I disagree it tells me to go away. No thanks.

I know what's happening. I want it to do things on my terms. It can suggest things, provide alternatives, but this refusal is extremely unhelpful.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#118

Post by iamjackg »

pera wrote: post #112
gavinray wrote: post #69

Earlier quoted context omitted.

Convert the GTK 3 and GTK 4 API documentation into a single `.txt` file each. Upload one of your platform-specific C++ file's source, along with the doc `.txt` into your LLM of choice. Either ask it for a conversion function-by-function, or separate it some other way logically such that the output doesn't get truncated. Would be surprised if this didn't work, to be honest.

Do you really need to provide the docs? I would have imagined that those docs are included in their training sets. There is even a guide on how to migrate from GTK3 to GTK4, so this seems to be a low-hanging fruit job for an LLM iff they are okay for coding.

You might not need to, but LLMs don't have perfect recall -- they're (variably) lossy by nature. Providing documentation is a pretty much universally accepted way to drastically improve their output.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#119

Post by HarHarVeryFunny »

I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivity tool when we move past demos and function writing to tackling real world projects.

It seems to me that where we are today, AI is only useful for coding for very localized tasks, and even there mostly where it's something commonplace and where the user knows enough to guide the AI when it's failing. I'm not at all convinced it's going to get much better until we have models that can actually learn (vs pre-trained) and are motivated to do so.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#120

Post by theshrike79 »

cjonas wrote: post #46
neal_ wrote: post #29

I was using gemini 2.5 pro yesterday and it does seem decent. I still think claude 3.5 is better at following instruction then the new 3.7 model which just goes ham messing stuff up. Really disappointed by Cursor and the Claude CLI tool, for me they create more problems then fix. I cant figure out how to use them on any of my projects with out them ruining the project and creating terrible tech debt. I really like th…

My only experience is via cursor but I'd agree in that context 3.7 is worse than 3.5. 3.7 goes crazy trying to fix any little linter errors and often gets confused and will just hammer away, making things worse until I stop generation. I think if I let it continue it would probably proposed rm -rf and start over at some point :). Again, this could just have to do with the way cursor is prompting it.

I asked claude 3.7 to move a perfectly working module to another location.

What did it do?

A COMPLETE FUCKING REWRITE OF THE MODULE.

The result did work, because of unit tests etc. but still, it has a habit of going down the rabbit hole of fixing and changing 42 different things when you ask for one change.

Post reply on HN