Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance • Hacker News Board

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#21

Post by lukehoban » Mon, Jun 09, 2025, 1:53 AM UTC

(Disclaimer: I work on coding agents at GitHub)

This data is great, and it is exciting to see the rapid growth of autonomous coding agents across GitHub.

One thing to keep in mind regarding merge rates is that each of these products creates the PR at a different phase of the work. So just tracking PR create to PR merge tells a different story for each product.

In some cases, the work to iterate on the AI generated code (and potentially abandon it if not sufficiently good) is done in private, and only pushed to a GitHub PR once the user decides they are ready to share/merge. This is the case for Codex for example. The merge rates for product experiences like this will look good in the stats presented here, even if many AI generated code changes are being abandoned privately.

For other product experiences, the Draft PR is generated immediately when a task is assigned, and users can iterate on this “in the open” with the coding agent. This creates more transparency into both the success and failure cases (including logs of the agent sessions for both). This is the case for GitHub Copilot coding agent for example. We believe this “learning in the open” is valuable for individuals, teams, and the industry. But it does lead to the merge rates reported here appearing worse - even if logically they are the same as “task assignment to merged PR” success rates for other tools.

We’re looking forward to continuing to evolve the notion of Draft PR to be even more natural for these use cases. And to enabling all of these coding agents to benefit from open collaboration on GitHub.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#22

Post by cap11235 » Mon, Jun 09, 2025, 1:55 AM UTC

a_bonobo wrote: post #18

dimitri-vs wrote: post #4

This might be an obvious questions but why is Claude Code not included?

I think the OP's page works because these coding agents identify themselves as the PR author so the creator can just search Github's issue tracker for things like is:pr+head:copilot or is:pr+head:codex It seems like Claude Code doesn't do that? some preliminary searching reveals that PRs generated by people using Claude Code use their own user account but may sign that they used Claude, example https://github.com/ant…

Claude does credit itself in the commit messages. eg:

feat: add progress bar for token probability calculation

- Add optional progress_cb parameter to get_token_probs function

- Integrate `rich` progress bar in CLI showing real-time token processing progress

- Add comprehensive tests for progress callback functionality

- Maintain backward compatibility with optional parameter

Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#23

Post by deadbabe » Mon, Jun 09, 2025, 1:56 AM UTC

osigurdson wrote: post #15

I've been underwhelmed with dedicated tools like Windsurf and Cursor in the sense that they are usually more annoying than just using ChatGPT. They have their niche but they are just so incredibly flow destroying it is hard to use them for long periods of time. I just started using Codex casually a few days ago though and already have 3 PRs. While different tools for different purposes make sense, Codex's fully async…

You can just use Cursor as a chat assistant if you want.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#24

Post by cjbarber » Mon, Jun 09, 2025, 2:00 AM UTC

Seems like the high order bit impacting results here might be how difficult the PR is?

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#25

Post by TZubiri » Mon, Jun 09, 2025, 2:03 AM UTC

Why is there 170k PR for a product released last month, but 700 for a product that has been around for like 6 months and was so popular it got acquired for 3B?

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#26

Post by a_bonobo » Mon, Jun 09, 2025, 2:16 AM UTC

cap11235 wrote: post #22

a_bonobo wrote: post #18

Earlier quoted context omitted.

I think the OP's page works because these coding agents identify themselves as the PR author so the creator can just search Github's issue tracker for things like is:pr+head:copilot or is:pr+head:codex It seems like Claude Code doesn't do that? some preliminary searching reveals that PRs generated by people using Claude Code use their own user account but may sign that they used Claude, example https://github.com/ant…

Claude does credit itself in the commit messages. eg: feat: add progress bar for token probability calculation - Add optional progress_cb parameter to get_token_probs function - Integrate `rich` progress bar in CLI showing real-time token processing progress - Add comprehensive tests for progress callback functionality - Maintain backward compatibility with optional parameter Generated with [Claude Code]( https://cla…

OK then OP can slightly change their site by using a different search term:

https://github.com/search?q=is:pr+is:merged+Co-Authored-By:+...

Instead of looking at the author of the PR, look for that 'Co-Authored-By: Claude' text bit.

That way I get 753 closed PRs and '1k' PRs in total, that's a pretty good acceptance rate.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#27

Post by soamv » Mon, Jun 09, 2025, 2:17 AM UTC

lukehoban wrote: post #21

(Disclaimer: I work on coding agents at GitHub) This data is great, and it is exciting to see the rapid growth of autonomous coding agents across GitHub. One thing to keep in mind regarding merge rates is that each of these products creates the PR at a different phase of the work. So just tracking PR create to PR merge tells a different story for each product. In some cases, the work to iterate on the AI generated co…

This is a great point! But there's an important tradeoff here about human engineering time versus the "learning in the open" benefits; a PR discarded privately consumes no human engineering time, a fact that the humans involved might appreciate. How do you balance that tradeoff? Is there such a thing as a diff that's "too bad" to iterate on with a human?

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#28

Post by elliotec » Mon, Jun 09, 2025, 2:20 AM UTC

osigurdson wrote: post #15

I've been underwhelmed with dedicated tools like Windsurf and Cursor in the sense that they are usually more annoying than just using ChatGPT. They have their niche but they are just so incredibly flow destroying it is hard to use them for long periods of time. I just started using Codex casually a few days ago though and already have 3 PRs. While different tools for different purposes make sense, Codex's fully async…

Have you tried Claude code? I’m surprised it’s not in this analysis but in my personal experience, the competition doesn’t even touch it. I’ve tried them all in earnest. My toolkit has been (neo)vim and tmux for at least a decade now so I understand the apprehension for less terminal-inclined folks that prefer other stuff but it’s my jam and just crushes it.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#29

Post by throwaway314155 » Mon, Jun 09, 2025, 2:27 AM UTC

Is this data not somewhat tainted by the fact that there's really zero way to identify how much a human was or wasn't "in the loop" before the PR was created?

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#30

Post by tptacek » Mon, Jun 09, 2025, 2:29 AM UTC

throwaway314155 wrote: post #29

Is this data not somewhat tainted by the fact that there's really zero way to identify how much a human was or wasn't "in the loop" before the PR was created?

I kind of wondered about that re: Devin vs. Cursor, because the people I know that happen to use Devin are also very hands-on with the code they end up merging.

But you could probably filter this a bit by looking at PR commit counts?