Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

composio.dev

Post reply on HN
Print view View original

251–260 of 336 posts

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#251

Post by qwertox »

stickfigure wrote: post #169
phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

My coding challenges are all variations on "start with this 1.5M line Spring project, full of multi-thousand-line files..."

But you are aware that their limited context length just won't be able to deal with this?

That's like saying that you're judging a sedan by its capability of performing the job of a truck.

Wait, you were being sarcastic?

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#252

Post by pdntspa »

ttul wrote: post #79
phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

Send the whole repo to AI Studio using my vibe coded tool `llm_globber` and let Gemini chew on it. You can get this done in a few hours.

Famous last words!

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#253

Post by stn8188 »

phkahler wrote: post #230
amelius wrote: post #143

Earlier quoted context omitted.

FWIW, what I want most in Solvespace is a way to do chamfers and fillets. And a way to define parameters (not sure if that's already possible).

>> FWIW, what I want most in Solvespace is a way to do chamfers and fillets. I've outlined a function for that and started to write the code. At a high level it's straight forward, but the details are complex. It'll probably be a year before it's done. >> And a way to define parameters (not sure if that's already possible). This is an active work in progress. A demo was made years ago, but it's buggy and incomplete.…

While I second the same request, I'm also incredibly grateful for Solvespace as a tool. It's my favorite MCAD program, and I always reach for it before any others. Thank you for your work on it!

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#254

Post by nowittyusername »

kaiokendev wrote: post #131
HarHarVeryFunny wrote: post #119

I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivi…

I made this NES emulator with Claude last week [0]. I'd say it was a pretty non-trivial task. It involved throwing a lot of NESDev docs, Disch mapper docs, and test rom output + assembly source code to the model to figure out. [0]: https://kaiokendev.github.io/nes/

I am considering training a custom Lora on atari roms and see if i could get a working game out of it with the Loras use. The thinking here is that atari, nes, snes, etc... roms are a lot smaller in size then a program that runs natively on whatever os. Lees lines of code to write for the LLM means less chance of a screw up. take the rom, convert it to assembly, perform very detailed captions on the rom and train.... if this works this would enable anyone to create games with one prompt which are a lot higher quality then the stuff being made now and with less complexity. If you made an emulator with the use of an llm, that means it understands assembly well enough so i think there might be hope for this idea.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#255

Post by stickfigure »

qwertox wrote: post #251
stickfigure wrote: post #169

Earlier quoted context omitted.

My coding challenges are all variations on "start with this 1.5M line Spring project, full of multi-thousand-line files..."

But you are aware that their limited context length just won't be able to deal with this? That's like saying that you're judging a sedan by its capability of performing the job of a truck. Wait, you were being sarcastic?

I am indeed saying that a sedan is incapable of handling my gigantic open-pit superfund site.

But I'll go a little farther - most meaningful, long-lived, financially lucrative software applications are metaphorically closer to the open-pit mine than the adorable backyard garden that AI tools can currently handle.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#256

Post by mvdtnz »

I must be missing something about Gemini. When I use the web UI it won't even let me upload source code files directly. If I manually copy some code into a directory and upload that I do get it to work, but the coding output is hilariously bad. It produces ludicrously verbose code that so far for me has been 200% wrong every time.

This is on a Gemini 2.5 Pro free trial. Also - god damn is it slow.

For context this is on a 15k LOC project built about 75% using Claude.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#257

Post by SpaceNoodled »

stavros wrote: post #248
SpaceNoodled wrote: post #243

Earlier quoted context omitted.

No, it's simply being demonstrated that they're not as useful as some claim.

By saying "why do I have to use a specific technique, instead of naively, to get what I want"?

"Why do I have to put in more work to use this tool vs. not using it?"

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#258

Post by ramesh31 »

semi-extrinsic wrote: post #184
ramesh31 wrote: post #84

Earlier quoted context omitted.

You guys really need a Docker build. This dependency chain with submodules is a nightmare.

Alternative perspective: you kids with your Docker builds need to roll up your sleeves and learn how to actually compile a semi-complicated project if you expect to be able to contribute back to said project.

>"Alternative perspective: you kids with your Docker builds need to roll up your sleeves and learn how to actually compile a semi-complicated project if you expect to be able to contribute back to said project."

Well, that attitude is probably why the issue has been open for 2 years.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#259

Post by phkahler »

snickell wrote: post #152
phkahler wrote: post #113

Earlier quoted context omitted.

>> This is the smoothest tom sawyer move I've ever seen IRL That made me laugh. True, but not really the motivation. I honestly don't think LLMs can code significant real-world things yet and I'm not sure how else to prove that since they can code some interesting things. All the talk about putting programmers out of work has me calling BS but also thinking "show me". This task seems like a good combination of simple…

Yes, very much agree, an interesting benchmark. Particularly because it’s in a “tier 2” framework (gtkmm) in terms of amount of code available to train an LLM on. That tests the LLMs ability to plan and problem solve compared with, say, “convert to the latest version of react” where the LLM has access to tens of thousands (more?) of similar ports in its training dataset and more has to pattern match.

>> Particularly because it’s in a “tier 2” framework (gtkmm) in terms of amount of code available to train an LLM on.

I asked GPT4 to write an empty GTK4 app in C++. I asked for a menu bar with File, Edit, View at the top and two GL drawing areas separated by a spacer. It produced what looked like usable code with a couple lines I suspected were out of place. I did not try to compile it so don't know if it was a hallucination, but it did seem to know about gtkmm 4.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#260

Post by ramesh31 »

phkahler wrote: post #240
ramesh31 wrote: post #84

Earlier quoted context omitted.

You guys really need a Docker build. This dependency chain with submodules is a nightmare.

I'm a hater of complexity and build systems in general. Following the instructions for building solvespace on Linux worked for me out of the box with zero issues and is not difficult. Just copy some commands: https://github.com/solvespace/solvespace?tab=readme-ov-file#...

>I'm a hater of complexity and build systems in general.

But you already have a complex cmake build system in place. Adding a standard Docker image with all the deps for devs to compile on would do nothing but make contributing easier, and would not affect your CI/CD/testing pipeline at all. I followed the readme and spent half an hour trying to get this to build for MacOS before giving up.

If building your project for all supported environments requires anything more than a single one-line command, you're doing it wrong.

Post reply on HN