Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

composio.dev

Post reply on HN
Print view View original

191–200 of 336 posts

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#191

Post by maxloh »

breadwinner wrote: post #190

The loser in the AI model competition appears to be... Microsoft. When ChatGPT was the only game in town Microsoft was seen as a leader, thanks to their wise investment in Open AI. They relied on Open AI's model and didn't develop their own. As a result Microsoft has no interesting AI products. Copilot is a flop. Bing failed to take advantage of AI, Perplexity ate their lunch. Satya Nadella last year: “Google should…

Note that Microsoft do have their own LLM team, and their own model called Phi-4.

https://huggingface.co/microsoft/phi-4

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#192

Post by stared »

Just a moment ago I tried to use Gemini 2.5 (in Cursor) to use Python Gemini SDK. It failed, even with a few iterations.

Then run Claude 3.7 - it worked fine.

So yeah, depends on the case. But I am surprised that model creators don't put extra effort into dealing with setting their own tools.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#193

Post by semi-extrinsic »

stavros wrote: post #180
cluckindan wrote: post #160

Earlier quoted context omitted.

I agree. I tried something similar: a conversion of a simple PHP library from one system to another. It was only like 500 loc but Gemini 2.5 completely failed around line 300, and even then its output contained straight up hallucinations, half-brained additions, wrong namespaces for dependencies, badly indented code and other PSR style violations. Worse, it also changed working code and broke it.

Try asking it to generate a high-level plan of how it's going to do the conversion first, then to generate function definitions for the new functions, then have it generate tests for the new functions, then actually write them, while giving it the output of the tests. It's not like people just one-shot a whole module of code, why would LLMs?

I know many people who can and will one-shot a rewrite of 500 LOC. In my world, 500 LOC is about the length of a single function. I don't understand why we should be talking about generating a high level plan with multiple tests etc. for a single function.

And I don't think this is uncommon. Just a random example from Github, this file is 1800 LOC and 4 functions. It implements one very specific thing that's part of a broader library. (I have no affiliation with this code.)

https://github.com/elemental/Elemental/blob/master/src/optim...

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#194

Post by stavros »

semi-extrinsic wrote: post #193
stavros wrote: post #180

Earlier quoted context omitted.

Try asking it to generate a high-level plan of how it's going to do the conversion first, then to generate function definitions for the new functions, then have it generate tests for the new functions, then actually write them, while giving it the output of the tests. It's not like people just one-shot a whole module of code, why would LLMs?

I know many people who can and will one-shot a rewrite of 500 LOC. In my world, 500 LOC is about the length of a single function. I don't understand why we should be talking about generating a high level plan with multiple tests etc. for a single function. And I don't think this is uncommon. Just a random example from Github, this file is 1800 LOC and 4 functions. It implements one very specific thing that's part of…

> I don't understand why we should be talking about generating a high level plan with multiple tests etc. for a single function.

You don't have to, you can write it by hand. I thought we were talking about how we can make computers write code, instead of humans, but it seems that we're trying to prove that LLMs aren't useful instead.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#195

Post by chrismorgan »

stavros wrote: post #180
cluckindan wrote: post #160

Earlier quoted context omitted.

I agree. I tried something similar: a conversion of a simple PHP library from one system to another. It was only like 500 loc but Gemini 2.5 completely failed around line 300, and even then its output contained straight up hallucinations, half-brained additions, wrong namespaces for dependencies, badly indented code and other PSR style violations. Worse, it also changed working code and broke it.

Try asking it to generate a high-level plan of how it's going to do the conversion first, then to generate function definitions for the new functions, then have it generate tests for the new functions, then actually write them, while giving it the output of the tests. It's not like people just one-shot a whole module of code, why would LLMs?

> It's not like people just one-shot a whole module of code, why would LLMs?

For conversions between languages or libraries, you often do just one-shot it, writing or modifying code from start to end in order.

I remember 15 years ago taking a 10,000 line Java code base and porting it to JavaScript mostly like this, with only a few areas requiring a bit more involved and non-sequential editing.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#197

Post by skydhash »

jchw wrote: post #165
phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

I suspect it probably won't work, although it's not necessarily because an LLM architecture could never perform this type of work, but rather because it works best when the training set contains inordinate sample data. I'm actually quite shocked at what they can do in TypeScript and JavaScript, but they're definitely a bit less "sharp" when it comes to stuff outside of that zone in my experience. The ridiculous amoun…

> the cynicism isn't out of fear or panic, its disappointment over and over and over

Very much this. When you criticize LLM's marketing, people will say you're a ludite.

I'd bet that no one actually likes to write code, as in typing into an editor. We know how to do it, and it's easy enough to enter in a flow state while doing it. But everyone is trying to write less code by themselves with the proliferation of reusable code, libraries, framework, code generators, metaprogramming,...

I'd be glad if I could have a DAW or CAD like interface with very short feedback (the closest is live programming with Smalltalk). So that I don't have to keep visualizing the whole project (it's mentally taxing).

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#198

Post by raffkede »

I had huge success letting Gemini 2.5 oneshot whole codebases in a single text file format and then split it up with a script. It's putting in work for like 5 minutes and spits out a working codebase, I also asked it to show of a little bit and it almost one shotted a java cloud service to generate pdf invoices from API calls, (made some minor mistakes but after feeding them back it fixed them)

I basically use two scripts one to flatten the whole codebase into one text file and one to split it, give it a shot it's amazing...

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#199

Post by pcwelder »

amazingamazing wrote: post #178
pcwelder wrote: post #92

Earlier quoted context omitted.

Gemini 2.5 pro hasn't been as good as Sonnet for me. The prompt I have tried repeatedly is creating a react-vite-todo app. It doesn't figure out tailwind related issues. Real chats: Gemini: https://github.com/rusiaaman/chat.md/blob/main/samples/vite-... Sonnet 3.7: https://github.com/rusiaaman/chat.md/blob/main/samples/vite-... Exact same settings, using MCP server for tool calling, using OpenAI api interface. PS: th…

your links don't work

The repo was private, updated. Thanks!!

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#200

Post by VladVladikoff »

maxloh wrote: post #191
breadwinner wrote: post #190

The loser in the AI model competition appears to be... Microsoft. When ChatGPT was the only game in town Microsoft was seen as a leader, thanks to their wise investment in Open AI. They relied on Open AI's model and didn't develop their own. As a result Microsoft has no interesting AI products. Copilot is a flop. Bing failed to take advantage of AI, Perplexity ate their lunch. Satya Nadella last year: “Google should…

Note that Microsoft do have their own LLM team, and their own model called Phi-4. https://huggingface.co/microsoft/phi-4

Recently I was looking for a small LLM that could perform reasonably well while answering questions with low latency, for near realtime conversations running on a single RTX 3090. I settled on Microsoft’s Phi-4 model so far. However I’m not sure yet if my choice is good and open to more suggestions!
Post reply on HN