Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

Post by SpaceNoodled » Mon, Mar 31, 2025, 7:23 PM UTC

stavros wrote: post #180

cluckindan wrote: post #160

Earlier quoted context omitted.

I agree. I tried something similar: a conversion of a simple PHP library from one system to another. It was only like 500 loc but Gemini 2.5 completely failed around line 300, and even then its output contained straight up hallucinations, half-brained additions, wrong namespaces for dependencies, badly indented code and other PSR style violations. Worse, it also changed working code and broke it.

Try asking it to generate a high-level plan of how it's going to do the conversion first, then to generate function definitions for the new functions, then have it generate tests for the new functions, then actually write them, while giving it the output of the tests. It's not like people just one-shot a whole module of code, why would LLMs?

Post by SpaceNoodled » Mon, Mar 31, 2025, 7:24 PM UTC

stavros wrote: post #194

semi-extrinsic wrote: post #193

Earlier quoted context omitted.

I know many people who can and will one-shot a rewrite of 500 LOC. In my world, 500 LOC is about the length of a single function. I don't understand why we should be talking about generating a high level plan with multiple tests etc. for a single function. And I don't think this is uncommon. Just a random example from Github, this file is 1800 LOC and 4 functions. It implements one very specific thing that's part of…

> I don't understand why we should be talking about generating a high level plan with multiple tests etc. for a single function. You don't have to, you can write it by hand. I thought we were talking about how we can make computers write code, instead of humans, but it seems that we're trying to prove that LLMs aren't useful instead.

Post by SpaceNoodled » Mon, Mar 31, 2025, 7:26 PM UTC

skydhash wrote: post #197

jchw wrote: post #165

Earlier quoted context omitted.

I suspect it probably won't work, although it's not necessarily because an LLM architecture could never perform this type of work, but rather because it works best when the training set contains inordinate sample data. I'm actually quite shocked at what they can do in TypeScript and JavaScript, but they're definitely a bit less "sharp" when it comes to stuff outside of that zone in my experience. The ridiculous amoun…

> the cynicism isn't out of fear or panic, its disappointment over and over and over Very much this. When you criticize LLM's marketing, people will say you're a ludite. I'd bet that no one actually likes to write code, as in typing into an editor. We know how to do it, and it's easy enough to enter in a flow state while doing it. But everyone is trying to write less code by themselves with the proliferation of reusa…

Post by [deleted] » Mon, Mar 31, 2025, 7:29 PM UTC

Philpax wrote: post #225

semi-extrinsic wrote: post #184

Earlier quoted context omitted.

Alternative perspective: you kids with your Docker builds need to roll up your sleeves and learn how to actually compile a semi-complicated project if you expect to be able to contribute back to said project.

If your project is hard to build, that's your problem, not mine. I'll simply spend my time working on projects that respect it.

Post by jay_kyburz » Mon, Mar 31, 2025, 7:34 PM UTC

jchw wrote: post #165

phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

I suspect it probably won't work, although it's not necessarily because an LLM architecture could never perform this type of work, but rather because it works best when the training set contains inordinate sample data. I'm actually quite shocked at what they can do in TypeScript and JavaScript, but they're definitely a bit less "sharp" when it comes to stuff outside of that zone in my experience. The ridiculous amoun…