Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

composio.dev

Post reply on HN
Print view View original

131–140 of 336 posts

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#131

Post by kaiokendev »

HarHarVeryFunny wrote: post #119

I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivi…

I made this NES emulator with Claude last week [0]. I'd say it was a pretty non-trivial task. It involved throwing a lot of NESDev docs, Disch mapper docs, and test rom output + assembly source code to the model to figure out.

[0]: https://kaiokendev.github.io/nes/

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#132

Post by diggan »

llm_nerd wrote: post #21
diggan wrote: post #8

Earlier quoted context omitted.

> has one million in context window Is this effective context window or just the absolute limit? A lot of the models that claim to support very large context windows cannot actually successfully do the typical "needle in a haystack" test, but I'm guessing there are published results somewhere demonstrating Gemini 2.5 Pro can actually find the needle?

Google has had almost perfect recall in the needle in the haystack test since 1.5[1], achieving close to 100% over the entire context window. I can't provide a link benchmarking 2.5 Pro in particular, but this has been a solved problem with Google models so I assume the same is true with their new model. [1] https://cloud.google.com/blog/products/ai-machine-learning/t...

Has those results been reproduced elsewhere with other benchmarks than what Google seems to use?

Hard to trust their own benchmarks at this point, and Im not home at the moment so cant try it myself either.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#133

Post by bix6 »

phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

Curious if you’ve tried this yourself yet? I’d love to see side by side of a human solo vs a human with copilot for something like this. AI will surely make mistakes so who will be faster / have better code in the end?

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#134

Post by eugenekolo »

It's definitely an attempt to compare models, and Gemini clearly won in the tests. But, I don't think the tests are particularly good or showcasing. It's generally an easy problem to ask AI to give you greenfields JS code for common tasks, and Leetcode's been done 1000 times on Github and stackoverflow, so the solutions are all right there.

I'd like to see tests that are more complicated for AI things like refactoring an existing codebase, writing a program to auto play God of War for you, improving the response time of a keyboard driver and so on.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#135

Post by Workaccount2 »

pera wrote: post #112
gavinray wrote: post #69

Earlier quoted context omitted.

Convert the GTK 3 and GTK 4 API documentation into a single `.txt` file each. Upload one of your platform-specific C++ file's source, along with the doc `.txt` into your LLM of choice. Either ask it for a conversion function-by-function, or separate it some other way logically such that the output doesn't get truncated. Would be surprised if this didn't work, to be honest.

Do you really need to provide the docs? I would have imagined that those docs are included in their training sets. There is even a guide on how to migrate from GTK3 to GTK4, so this seems to be a low-hanging fruit job for an LLM iff they are okay for coding.

LLMs are not data archives. They are god awful at storing data, and even calling them a lossy compression tool is a stretch because it implies they are a compression tool for data.

LLM's will always benefit from in context learning because they don't have a huge archive of data to draw on (and even when they do, they are not the best at selecting data to incorporate).

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#136

Post by skerit »

I've been using Gemini 2.5 Pro with Roo-Code a lot these past few days. It has really helped me a lot. I managed to get it to implemented entire features. (With some manual cleaning up at the end)

The fact that it's free for now (I know they use it for training, that's OK) is a big plus, because I've had to restart a task from scratch quite a few time. If I calculate what this would have cost me using Claude, it would have been 200-300 euros.

I've noticed that as soon as it makes a mistake (messing up the diff format is a classic), the current task is basically a total loss. For some reason, most coding tools basically just inform the model they made a mistake and should try again... but at that point, it's broken response is part of the history, and it's basically multi-shotting itself into making more mistakes. They should really just filter these out.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#137

Post by georgemcbay »

kordlessagain wrote: post #122
phkahler wrote: post #55

Here is a real coding problem that I might be willing to make a cash-prize contest for. We'd need to nail down some rules. I'd be shocked if any LLM can do this: https://github.com/solvespace/solvespace/issues/1414 Make a GTK 4 version of Solvespace. We have a single C++ file for each platform - Windows, Mac, and Linux-GTK3. There is also a QT version on an unmerged branch for reference. The GTK3 file is under 2KLOC.…

What’s the point of a one-to-one GTK3 → GTK4 rewrite when the user experience doesn’t improve at all? Why not modularize the backend and build a better UI with tech that’s actually relevant in 2025?

I'm not the person you are asking but the point of this whole thing seems to be as a test for how possible it is for an LLM to 'vibe code' a port of this nature and not really because they care that much about a port existing.

The fact that they haven't done the port in the normal way suggests they basically agree with what you said here (not worth the ROI), but hey if you can get the latest AI code editor to spit out a perfectly working port in minutes, why not?

FWIW, my assessment of LLMs is the same as theirs. The hype is far greater than the practical usefulness, and I say this as someone who is using LLMs pretty regularly now.

They aren't useless, but the idea that they will be writing 90% of our code soon is just completely at odds with my day to day experience getting them to do actual specific tasks rather than telling them to "write Tetris for XYZ" and blog about how great they are because it produced something roughly what I asked for without much specificity.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#138

Post by martin-t »

neal_ wrote: post #29

I was using gemini 2.5 pro yesterday and it does seem decent. I still think claude 3.5 is better at following instruction then the new 3.7 model which just goes ham messing stuff up. Really disappointed by Cursor and the Claude CLI tool, for me they create more problems then fix. I cant figure out how to use them on any of my projects with out them ruining the project and creating terrible tech debt. I really like th…

Whenever I read about LLMs or try to use them, I feel like I am asleep in a dream where two contradicting things can be true at the same time.

On one hand, you have people claiming "AI" can now do SWE tasks which take humans 30 minutes or 2 hours and the time doubles every X months so by Y year, SW development will be completely automated.

On the other hand, you have people saying exactly what you are saying. Usually that LLMs have issues even with small tasks and that repeated/prolonged use generates tech debt even if they succeed on the small tasks.

These 2 views clearly can't both be true at the same time. My experience is the second category so I'd like to chalk up the first as marketing hype but it's confusing how many people who have seemingly nothing to gain from the hype contribute to it.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#139

Post by jstummbillig »

This has not been my experience using it with Windsurf, which touches on an interesting point: When a tool has been optimized around one model, how much is it inhibiting another (newly released) model and how much adjustment is required to take advantage of the new model? Increasingly, as tools get better, we will not directly interact with the models. I wonder how the tool makers handle this.

Re: Gemini 2.5 Pro vs. Claude 3.7 Sonnet: Coding Comparison

#140

Post by lordswork »

HarHarVeryFunny wrote: post #119

I'd like to see an honest attempt by someone to use one of these SOTA models to code an entire non-trivial app. Not a "vibe coding" flappy bird clone or minimal ioS app (call API to count calories in photo), but something real - say 10K LOC type of complexity, using best practices to give the AI all the context and guidance necessary. I'm not expecting the AI to replace the programmer - just to be a useful productivi…

I'm at 3k LOC on a current Rust project I'm mostly vibe coding with my very limited free time. Will share when I hit 10k :)
Post reply on HN