Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

aavetis.github.io

Post reply on HN
Print view View original

1–10 of 124 posts

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#3

Post by frognumber »

Missing data: I don't make a codex PR if it's nonsense.

Poor data: If I make one, I either if I want to:

a) Merge it (success)

b) Modify it (sometimes success, sometimes not). In one case, Codex made the wrong changes in all the right places, but it was still easier to work from that by hand.

c) Pick ideas from it (partial success)

So simple merge rates don't say much.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#5

Post by csallen »

dimitri-vs wrote: post #4

This might be an obvious questions but why is Claude Code not included?

I believe these are all "background" agents that, by default, are meant to write code and issue pull requests without you watching/babysitting/guiding the process. I haven't used Claude Code in a while, but from what I recall, it's not that.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#7

Post by tmvnty »

Merge rates is definitely a useful signal, but there are certainly other factors we need consider (PR small/big edits, refactors vs deps upgrades, direct merges, follow up PRs correcting merged mistakes, how easy it is to setup these AI agents, marketing, usage fees etc). Similar to how NPM downloads alone don’t necessarily reflect a package’s true success or quality.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#8

Post by koakuma-chan »

csallen wrote: post #5
dimitri-vs wrote: post #4

This might be an obvious questions but why is Claude Code not included?

I believe these are all "background" agents that, by default, are meant to write code and issue pull requests without you watching/babysitting/guiding the process. I haven't used Claude Code in a while, but from what I recall, it's not that.

Claude Code can run in background and I don't see why it wouldn't be able to create pull requests if you gave it such a tool.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#9

Post by cap11235 »

csallen wrote: post #5
dimitri-vs wrote: post #4

This might be an obvious questions but why is Claude Code not included?

I believe these are all "background" agents that, by default, are meant to write code and issue pull requests without you watching/babysitting/guiding the process. I haven't used Claude Code in a while, but from what I recall, it's not that.

If you enable it in permissions, Claude is very happy to do so. For personal fun/experimental projects (usually I give it arXiv papers to implement), I generally have a couple Claude instances (on different projects) just chugging along all day. I have them write really detailed plans at the start (50-100 steps in the implementation plan, plus actual specifications for project structure, dev practices, and what the actual goals are). I iterate on these plan documents by having Claude write QUESTIONS.md which has dev questions for me to clarify, which I fill out with answers, and then instruct Claude to update the plan docs with my answers. Then most of my interaction throughout the day is just saying something like "P18" to implement implementation plan step #18. I instruct it in CLAUDE.md to stop after each step, output what automated tests have been written for P18's features, and I require that the LLM write a demo script that I can run that shows the features, using real APIs. I'm having a great time with it.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#10

Post by ilteris »

cap11235 wrote: post #9
csallen wrote: post #5

Earlier quoted context omitted.

I believe these are all "background" agents that, by default, are meant to write code and issue pull requests without you watching/babysitting/guiding the process. I haven't used Claude Code in a while, but from what I recall, it's not that.

If you enable it in permissions, Claude is very happy to do so. For personal fun/experimental projects (usually I give it arXiv papers to implement), I generally have a couple Claude instances (on different projects) just chugging along all day. I have them write really detailed plans at the start (50-100 steps in the implementation plan, plus actual specifications for project structure, dev practices, and what the a…

How much do you pay monthly? What kind of service do you use thanks
Post reply on HN