Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

composio.dev

Post reply on HN
Print view View original

171–180 of 336 posts

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#171

Post by redox99 »

HarHarVeryFunny wrote: post #119

I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivi…

I use cursor agent mode with claude on my NextJS frontend and Typescript GraphQL backend. It's a real, reasonably sized, production app that's a few years old (pre-ChatGPT).

I vibe code the vast majority features nowadays. I generally don't need to write a single line of code. It often makes some mistakes but the agent figures out that the tests fail, or it doesn't build, fixes it, and basically "one shots" it after it doing its thing.

Only occasionally I need to write a few lines of code or give it a hint when it gets stuck. But 99% of the code is written by cursor.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#172

Post by iamleppert »

phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

GTK is an abomination of a UI framework. You should be looking for another way to manage your UI entirely, not trying to keep up with the joneses, who will no doubt release something new in short order and set yet another hoop to jump through, without providing any benefit to you at all.

It's openly hostile to not consider the upgrade path of existing users, and make things so difficult that it requires huge lifts just to upgrade versions of something like a UI framework.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#173

Post by rom16384 »

bratao wrote: post #5

From my use case, the Gemini 2.5 is terrible. I have a complex Cython code in a single file (1500 lines) for a Sequence Labeling. Claude and o3 are very good in improving this code and following the commands. The Gemini always try to do unrelated changes. For example, I asked, separately, for small changes such as remove this unused function, or cache the arrays indexes. Every time it completely refactored the code a…

You can fix this using a system prompt to force it to reply just with a diff. It makes the generation much faster and much less prone to changing unrelated lines. Also try reducing the temperature to 0.4 for example, I find the default temperature of 1 too high. For sample system prompts see Aider Chat: https://github.com/Aider-AI/aider/blob/main/aider/coders/edi...

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#174

Post by aleph_minus_one »

pdntspa wrote: post #159
aleph_minus_one wrote: post #149

Earlier quoted context omitted.

> Why not modularize the backend and build a better UI with tech that’s actually relevant in 2025? Doing the second part is to my understanding actually the purpose of the stated task.

Why are you calling GTK4 irrelevant? Large swaths of Linux run on it and GTK3

> Why are you calling GTK4 irrelevant?

Quite the opposite: Gtk4 is relevant, and porting Solvespace to this relevant toolkit is the central part of the stated task.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#175

Post by sxp »

One prompt I use for testing is: "Using three.js, render a spinning donut with gl.TRIANGLE_STRIP". The catch here is that three.js doesn't support TRIANGLE_STRIP for architectural reasons[1]. Before I knew this, I got confused as to why all the AIs kept failing and gaslighting me about using TRIANGLE_STRIP. If the AI fails to tell the user that this is an impossible task, then it has failed the test. So far, I haven't found an AI that can determine that the request isn't valid.

[1] https://discourse.threejs.org/t/is-there-really-no-way-to-us...

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#176

Post by HarHarVeryFunny »

Pannoniae wrote: post #168
HarHarVeryFunny wrote: post #119

I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivi…

I've been using Claude 3.7 for various things, including helping in game development tasks. The generated code usually requires editing and it can't do autonomously more than a few functions at once but it's a fairly useful tool in terms of productivity. And the logic part is also quite good, can design out various ideas/algorithms, and suggest some optimisations. Tech stack is nothing fancy/rare but not the usual Re…

Interesting - thanks! This isn't the type of tech stack where I'd have expected it to do very well, so the fact that you're at least finding it to be productive is encouraging, although the (only) "function level competency" is similar to what I've experienced - enough to not have been encouraged to try anything more complex.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#177

Post by HarHarVeryFunny »

redox99 wrote: post #171
HarHarVeryFunny wrote: post #119

I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivi…

I use cursor agent mode with claude on my NextJS frontend and Typescript GraphQL backend. It's a real, reasonably sized, production app that's a few years old (pre-ChatGPT). I vibe code the vast majority features nowadays. I generally don't need to write a single line of code. It often makes some mistakes but the agent figures out that the tests fail, or it doesn't build, fixes it, and basically "one shots" it after…

That's pretty impressive - a genuine real-world use case where the AI is doing the vast majority of the work.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#178

Post by amazingamazing »

pcwelder wrote: post #92
amazingamazing wrote: post #23

In before people post contradictory anecdotes. It would be more helpful if people posted the prompt, and the entire context, or better yet the conversation, so we can all judge for ourselves.

Gemini 2.5 pro hasn't been as good as Sonnet for me. The prompt I have tried repeatedly is creating a react-vite-todo app. It doesn't figure out tailwind related issues. Real chats: Gemini: https://github.com/rusiaaman/chat.md/blob/main/samples/vite-... Sonnet 3.7: https://github.com/rusiaaman/chat.md/blob/main/samples/vite-... Exact same settings, using MCP server for tool calling, using OpenAI api interface. PS: th…

your links don't work

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#179

Post by dagw »

pera wrote: post #112
gavinray wrote: post #69

Earlier quoted context omitted.

Convert the GTK 3 and GTK 4 API documentation into a single `.txt` file each. Upload one of your platform-specific C++ file's source, along with the doc `.txt` into your LLM of choice. Either ask it for a conversion function-by-function, or separate it some other way logically such that the output doesn't get truncated. Would be surprised if this didn't work, to be honest.

Do you really need to provide the docs? I would have imagined that those docs are included in their training sets. There is even a guide on how to migrate from GTK3 to GTK4, so this seems to be a low-hanging fruit job for an LLM iff they are okay for coding.

Feeding them the docs makes a huge difference in my experience. The docs might be somewhere in the training set, but telling the LLM explicitly "Use these docs before anything else" solves a lot of problems the the LLM mixing up different versions of a library or confusing two different libraries with a similar API.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#180

Post by stavros »

cluckindan wrote: post #160
phkahler wrote: post #113

Earlier quoted context omitted.

>> This is the smoothest tom sawyer move I've ever seen IRL That made me laugh. True, but not really the motivation. I honestly don't think LLMs can code significant real-world things yet and I'm not sure how else to prove that since they can code some interesting things. All the talk about putting programmers out of work has me calling BS but also thinking "show me". This task seems like a good combination of simple…

I agree. I tried something similar: a conversion of a simple PHP library from one system to another. It was only like 500 loc but Gemini 2.5 completely failed around line 300, and even then its output contained straight up hallucinations, half-brained additions, wrong namespaces for dependencies, badly indented code and other PSR style violations. Worse, it also changed working code and broke it.

Try asking it to generate a high-level plan of how it's going to do the conversion first, then to generate function definitions for the new functions, then have it generate tests for the new functions, then actually write them, while giving it the output of the tests.

It's not like people just one-shot a whole module of code, why would LLMs?

Post reply on HN