Live data from Hacker News

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

composio.dev

301–310 of 336 posts

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#301
post #136

I've been using Gemini 2.5 Pro with Roo-Code a lot these past few days. It has really helped me a lot. I managed to get it to implemented entire features. (With some manual cleaning up at the end) The fact that it's free for now (I know they use it for training, that's OK) is a big plus, because I've had to restart a task from scratch quite a few time. If I calculate what this would have cost me using Claude, it woul…

Try this and watch it supercharge! https://docs.roocode.com/features/boomerang-tasks/

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#302

Every test task, including the coding test, is a greenfield project. Everything I would consider using LLMs for is not. Like, I would always need it to do some change or fix on a (large) existing project. Hell, even the examples that were generated would likely need subsequent alterations (ten times more effort goes into maintaining a line of code than writing it). So these tests are meaningless to me, as a measure o…

Indeed, I surprised to see that is has been in top-10 on HN for today. I thought everyone already realized that all of those examples like "create a flappy bird game" are not realistic and do not reflect the actual usefulness of the model, very few professionals in the industry endlessly create flappy bird games for a living.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#303
post #112

Earlier quoted context omitted.

Convert the GTK 3 and GTK 4 API documentation into a single `.txt` file each. Upload one of your platform-specific C++ file's source, along with the doc `.txt` into your LLM of choice. Either ask it for a conversion function-by-function, or separate it some other way logically such that the output doesn't get truncated. Would be surprised if this didn't work, to be honest.

Do you really need to provide the docs? I would have imagined that those docs are included in their training sets. There is even a guide on how to migrate from GTK3 to GTK4, so this seems to be a low-hanging fruit job for an LLM iff they are okay for coding.

Docs make them hallucinate a lot less. Unfortunately, all those docs will eat up the context window. Claude has "projects" for uploading them and Gemini2.5+ just has a very large window so maybe that's ok.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#304
post #262

Earlier quoted context omitted.

As someone who just adopted Cursor (and MCP) 2-3 weeks ago, Aider seems like a different world. The examples of "create a new simple video game" cause me to glaze over. Do you have a screencast of how you use aider to develop aider? I'd love to see how a savvy expert uses these tools for real-world solutions.

I actually get asked for screencasts a lot, so I made recently made some [0]. The recording of adding support for 100+ new coding languages with tree-sitter [1] shows some pretty advanced usage. It includes using aider to script downloading a collection of files, and using ad-hoc bash scripts to have aider modify a collection of files. [0] https://aider.chat/docs/recordings/ [1] https://aider.chat/docs/recordings/tre…

This is excellent, thank you.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#305
post #165

Earlier quoted context omitted.

I suspect it probably won't work, although it's not necessarily because an LLM architecture could never perform this type of work, but rather because it works best when the training set contains inordinate sample data. I'm actually quite shocked at what they can do in TypeScript and JavaScript, but they're definitely a bit less "sharp" when it comes to stuff outside of that zone in my experience. The ridiculous amoun…

> the cynicism isn't out of fear or panic, its disappointment over and over and over Very much this. When you criticize LLM's marketing, people will say you're a ludite. I'd bet that no one actually likes to write code, as in typing into an editor. We know how to do it, and it's easy enough to enter in a flow state while doing it. But everyone is trying to write less code by themselves with the proliferation of reusa…

> no one actually likes to write code

between this and..

> But everyone is trying to write less code by themselves with the proliferation of reusable code, libraries, framework, code generators, metaprogramming

.. this, is a massive gap. Personally speaking, I hate writing boilerplate code, y'know, old school Java with design patterns getter/setter, redundant multi-layer catch blocks, stateful for loops etc. That gets on my nerves, because it increases my work for little benefits. Cue modern coding practices and I'm almost exclusively thinking how to design solution to the problem at hand, and almost all the code is business logic exclusive.

This is where a lot of LLMs just fail. Handholding them all the way to correct solution feels like writing boilerplate again, except worse because I don't know when I'll be done. It doesn't help that most code available for LLMs is JS/TS/Java where boilerplate galore, but somehow I doubt giving them exclusively good codebases will help.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#306

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

> I'm not going to hold my breath.

The snark and pessimism nerd-sniped me :)

I've used AI heavily to maintain a cross-platform wrapper around llama.cpp. I figure its worth a shot.

I took a look and wanted to try but hit several hard blocks right away.

- There is no gtk-4 branch :o (presuming branch = git branch...Perhaps this is some project-specific terminology for a set of flags or something, and that's why I can't find it?)

- There's some indicators it is blocked by wxWidgets requiring GTK-4 support, which sounds much larger scope than advertised -- am I misunderstanding?

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#307

I remember back in the day when I did Visual Basic in the 90s there were a lot of cool "New Project from Template" things in Visual Studio, especially when you installed new frameworks and SDKs and stuff like that. With a click of a button you had something that kind of looked like a professional app! Or even now, the various create-whatever-app tooling in npm and node keeps on that legacy. Anyway, AI "coding" makes…

Visual basic created a revolution in software world especially for poor countries like India. You will be surprised how many systems were automated and turned into software driven processes. It was just mindblowing.

If AI driven software can do it on steroid it would be a massive impact on economy.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#308

Earlier quoted context omitted.

Yes, very much agree, an interesting benchmark. Particularly because it’s in a “tier 2” framework (gtkmm) in terms of amount of code available to train an LLM on. That tests the LLMs ability to plan and problem solve compared with, say, “convert to the latest version of react” where the LLM has access to tens of thousands (more?) of similar ports in its training dataset and more has to pattern match.

>> Particularly because it’s in a “tier 2” framework (gtkmm) in terms of amount of code available to train an LLM on. I asked GPT4 to write an empty GTK4 app in C++. I asked for a menu bar with File, Edit, View at the top and two GL drawing areas separated by a spacer. It produced what looked like usable code with a couple lines I suspected were out of place. I did not try to compile it so don't know if it was a hall…

It definitely knows what GTK4 is, when it freaked out on me and lost the code, it was using all gtkmm-4.0 headers, and had the compiler error count down to 10 (most likely with tons of logic errors, but who knows).

But LLMs performance varies (and this is a huge critique!) not just on what they theoretically know, but how, erm, cross-linked it is with everything else, and that requires lots of training data in the topic.

Metaphorically, I think this is a little like the difference for humans in math between being able to list+define techniques to solve integrals vs being able to fluidly apply them without error.

I think a big and very valid critique of LLMs (compared to humans) is that they are stronger at "memory" than reasoning. They use their vast memory as a crutch to hide the weaknesses in their reasoning. This makes benchmarks like "convert from gtkmm3 to gtkmm4" both challenging AND very good benchmarks of what real programmers are able to do.

I suspect if we gave it a similarly sized 2kloc conversion problem with a popular web framework in TS or JS, it would one-shot it. But again, its "cheating" to do this, its leveraging having read a zillion conversion by humans and what they did.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#309

Earlier quoted context omitted.

Try asking it to generate a high-level plan of how it's going to do the conversion first, then to generate function definitions for the new functions, then have it generate tests for the new functions, then actually write them, while giving it the output of the tests. It's not like people just one-shot a whole module of code, why would LLMs?

> It's not like people just one-shot a whole module of code, why would LLMs? For conversions between languages or libraries, you often do just one-shot it, writing or modifying code from start to end in order. I remember 15 years ago taking a 10,000 line Java code base and porting it to JavaScript mostly like this, with only a few areas requiring a bit more involved and non-sequential editing.

So, you didn't test it until the end? or did you have to build it in such a way that is was partially testable?

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#310
I really wish people would stop evaluating a model's coding capability with one-shots.

The vast majority of coding energy is what comes next.

Even today, sonnet-3.5 is still the best "existing code base" model. Which is gratifying (to Anthropic) and/or alarming to everyone else

Post reply on HN