Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

aavetis.github.io

Post reply on HN
Print view View original

31–40 of 124 posts

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#31

Post by ambicapter »

soamv wrote: post #27
lukehoban wrote: post #21

(Disclaimer: I work on coding agents at GitHub) This data is great, and it is exciting to see the rapid growth of autonomous coding agents across GitHub. One thing to keep in mind regarding merge rates is that each of these products creates the PR at a different phase of the work. So just tracking PR create to PR merge tells a different story for each product. In some cases, the work to iterate on the AI generated co…

This is a great point! But there's an important tradeoff here about human engineering time versus the "learning in the open" benefits; a PR discarded privately consumes no human engineering time, a fact that the humans involved might appreciate. How do you balance that tradeoff? Is there such a thing as a diff that's "too bad" to iterate on with a human?

Do people where you work spend time reviewing draft PRs? I wouldn’t do that unless asked to by the author.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#32

Post by ofou »

a_bonobo wrote: post #26
cap11235 wrote: post #22

Earlier quoted context omitted.

Claude does credit itself in the commit messages. eg: feat: add progress bar for token probability calculation - Add optional progress_cb parameter to get_token_probs function - Integrate `rich` progress bar in CLI showing real-time token processing progress - Add comprehensive tests for progress callback functionality - Maintain backward compatibility with optional parameter Generated with [Claude Code]( https://cla…

OK then OP can slightly change their site by using a different search term: https://github.com/search?q=is:pr+is:merged+Co-Authored-By:+... Instead of looking at the author of the PR, look for that 'Co-Authored-By: Claude' text bit. That way I get 753 closed PRs and '1k' PRs in total, that's a pretty good acceptance rate.

I'd submit a PR with this idea to improve coverage of agents

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#34

Post by thorum »

throwaway314155 wrote: post #29

Is this data not somewhat tainted by the fact that there's really zero way to identify how much a human was or wasn't "in the loop" before the PR was created?

With Jules, I almost always end up making significant changes before approving the PR. So “successful merge” is not great indicator of how well the model did in my case. I’ve merged PRs that were initially terrible after going in and fixing all the mistakes.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#35

Post by cap11235 »

elliotec wrote: post #28
osigurdson wrote: post #15

I've been underwhelmed with dedicated tools like Windsurf and Cursor in the sense that they are usually more annoying than just using ChatGPT. They have their niche but they are just so incredibly flow destroying it is hard to use them for long periods of time. I just started using Codex casually a few days ago though and already have 3 PRs. While different tools for different purposes make sense, Codex's fully async…

Have you tried Claude code? I’m surprised it’s not in this analysis but in my personal experience, the competition doesn’t even touch it. I’ve tried them all in earnest. My toolkit has been (neo)vim and tmux for at least a decade now so I understand the apprehension for less terminal-inclined folks that prefer other stuff but it’s my jam and just crushes it.

Right, after the Sonnet 4 release it was the first time I could tell an agent something and just let it run comfortably. As for the tool itself, I think a large part of its ability comes from how it writes recursive todo-lists for itself, which are shown to the user, so you can intervene early on the occasions it goes full Monkey's Paw.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#36

Post by threeseed »

deadbabe wrote: post #23
osigurdson wrote: post #15

I've been underwhelmed with dedicated tools like Windsurf and Cursor in the sense that they are usually more annoying than just using ChatGPT. They have their niche but they are just so incredibly flow destroying it is hard to use them for long periods of time. I just started using Codex casually a few days ago though and already have 3 PRs. While different tools for different purposes make sense, Codex's fully async…

You can just use Cursor as a chat assistant if you want.

But then you're paying far more than just using Claude web which can be used for tasks other than coding.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#38

Post by drawnwren »

ambicapter wrote: post #31
soamv wrote: post #27

Earlier quoted context omitted.

This is a great point! But there's an important tradeoff here about human engineering time versus the "learning in the open" benefits; a PR discarded privately consumes no human engineering time, a fact that the humans involved might appreciate. How do you balance that tradeoff? Is there such a thing as a diff that's "too bad" to iterate on with a human?

Do people where you work spend time reviewing draft PRs? I wouldn’t do that unless asked to by the author.

It’s hard enough for me to get time to review actual PRs, who are these engineers trawling through the drafts?

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#39

Post by simoncion »

TZubiri wrote: post #25

Why is there 170k PR for a product released last month, but 700 for a product that has been around for like 6 months and was so popular it got acquired for 3B?

It might be the case that "number of PRs" is roughly as good a metric as "number of lines of code produced".
Post reply on HN