Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

aavetis.github.io

Post reply on HN
Print view View original

101–110 of 124 posts

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#101

Post by softwaredoug »

jillesvangurp wrote: post #97
osigurdson wrote: post #15

I've been underwhelmed with dedicated tools like Windsurf and Cursor in the sense that they are usually more annoying than just using ChatGPT. They have their niche but they are just so incredibly flow destroying it is hard to use them for long periods of time. I just started using Codex casually a few days ago though and already have 3 PRs. While different tools for different purposes make sense, Codex's fully async…

OpenAI nailed the UX/DX with codex. This completely obsoletes cursor and similar IDEs. I don't need AI in my tools. I just need somebody to work on my code in parallel to me. I'm happy to interact via pull requests and branches. I found out that I have access to codex on Thursday with my plus subscription. I've created and merged about a dozen PRs with it on my OSS projects since then. It's not flawless but it's pret…

I love using codex to just explore code instead of searching. It’s a great tool to learn or research what’s happening in the code with great code breadcrumbs to find what you need to know

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#102

Post by cap11235 »

falcor84 wrote: post #89
cap11235 wrote: post #46

Earlier quoted context omitted.

Feels like a sort of pollution.

Why? That is its intent - unlike an IDE, it is intended to work autonomously and only get back to you after it has prepared the full changeset - which at that point you'd review via a PR. Where's the pollution in that?

The hundreds of thousands of commits?

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#103

Post by blagie »

polskibus wrote: post #57
lukehoban wrote: post #21

(Disclaimer: I work on coding agents at GitHub) This data is great, and it is exciting to see the rapid growth of autonomous coding agents across GitHub. One thing to keep in mind regarding merge rates is that each of these products creates the PR at a different phase of the work. So just tracking PR create to PR merge tells a different story for each product. In some cases, the work to iterate on the AI generated co…

What is your team’s take on the copyright for commits generated by ai agent ? Would the copyright protect it? Current US stance seems to be: https://www.copyright.gov/newsnet/2025/1060.html “It concludes that the outputs of generative AI can be protected by copyright only where a human author has determined sufficient expressive elements”. If entire commit is generated by AI then it is obvious what created it - it’s…

> If entire commit is generated by AI then it is obvious what created it - it’s AI.

This is not the case. The output of a compiler is 100% created by a compiler too. Copyright is based on where the creative aspect comes from.

I have had very little luck having 2025-era AIs manage the creative aspects of coding -- design, architecture, and similar -- and that's doubly true for what appears to be the relatively simplistic model in codex (as far as I can tell, codex trades off model complexity for model time; the model does a massive amount of work for a relatively small change).

However, it is much better than I am at the mechanical aspects. LLMs can fix mechanical bugs almost instantly (the sort of thing with a cut-and-paste fix in some build process from Stack Overflow), and generate massive amounts of code without typos or shallow bugs.

A good analogy is working with powertools versus handtools. I can do much more in one step, but I'm still in creative control.

The codebase I'm working on is pretty sophisticated, and I might imagine they could implement more cookiecutter things (e.g. a standard oauth workflow) more automatically.

However, even there -- or in discussions with larger models about my existing codebase -- what they do is in part based their creativity on human contributions to their training set. I'm not sure how to weigh that. An LLM oauth workflow might be considered the creative median of a lot of human-written code.

I write a lot of AGPL code, and at least in the 3.5 era, they were clearly trained on my code, and would happily print it out more-or-less verbatim. Indeed, it was to the point where I complained to OpenAI about it at the time, but never got a response. I suspect a lot of generated code will include some fractional contribution from me now (an infinitesimal fraction most of the time, but more substantial for niche code similar to my codebase).

So in generated code, we have a mixture of at least a few different pieces:

- User's contributions, in prompt, review, etc.

- Machine contributions

- Training set contributions

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#104

Post by jegudiel »

polskibus wrote: post #57
lukehoban wrote: post #21

(Disclaimer: I work on coding agents at GitHub) This data is great, and it is exciting to see the rapid growth of autonomous coding agents across GitHub. One thing to keep in mind regarding merge rates is that each of these products creates the PR at a different phase of the work. So just tracking PR create to PR merge tells a different story for each product. In some cases, the work to iterate on the AI generated co…

What is your team’s take on the copyright for commits generated by ai agent ? Would the copyright protect it? Current US stance seems to be: https://www.copyright.gov/newsnet/2025/1060.html “It concludes that the outputs of generative AI can be protected by copyright only where a human author has determined sufficient expressive elements”. If entire commit is generated by AI then it is obvious what created it - it’s…

AI Code and Copyright - Risky Business or Creative Power-Up(AI Generated Podcast)

https://open.spotify.com/episode/6o2Ik3w6c4x4DYILXwRSos?si=5...

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#105

Post by jegudiel »

polskibus wrote: post #57
lukehoban wrote: post #21

(Disclaimer: I work on coding agents at GitHub) This data is great, and it is exciting to see the rapid growth of autonomous coding agents across GitHub. One thing to keep in mind regarding merge rates is that each of these products creates the PR at a different phase of the work. So just tracking PR create to PR merge tells a different story for each product. In some cases, the work to iterate on the AI generated co…

What is your team’s take on the copyright for commits generated by ai agent ? Would the copyright protect it? Current US stance seems to be: https://www.copyright.gov/newsnet/2025/1060.html “It concludes that the outputs of generative AI can be protected by copyright only where a human author has determined sufficient expressive elements”. If entire commit is generated by AI then it is obvious what created it - it’s…

An unconventional license for AI-generated code. Maybe public domain, maybe not. Use freely, vibe responsibly.

https://jilvin.github.io/vibe-license/

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#106

Post by falcor84 »

cap11235 wrote: post #102
falcor84 wrote: post #89

Earlier quoted context omitted.

Why? That is its intent - unlike an IDE, it is intended to work autonomously and only get back to you after it has prepared the full changeset - which at that point you'd review via a PR. Where's the pollution in that?

The hundreds of thousands of commits?

Commits are pollution? Are you concerned that we'll run out of SHA-1 digests?

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#107

Post by chiph »

IanCal wrote: post #80
polskibus wrote: post #57

Earlier quoted context omitted.

What is your team’s take on the copyright for commits generated by ai agent ? Would the copyright protect it? Current US stance seems to be: https://www.copyright.gov/newsnet/2025/1060.html “It concludes that the outputs of generative AI can be protected by copyright only where a human author has determined sufficient expressive elements”. If entire commit is generated by AI then it is obvious what created it - it’s…

>If entire commit is generated by AI then it is obvious what created it - it’s AI. Whether it's committed or not is irrelevant to the conclusion there, the question is what was the input .

For something like a compiler where the output is mostly deterministic[0] I agree. For an AI that was trained on an unknown corpus, and that corpus changes over time, the output is much less deterministic and I would say you lose the human element needed of copyright claims.

If it can be shown that for the same prompt, run through the AI several times over perhaps a year, results in the same output - then I will change my mind. Or if the AI achieves personhood.

[0] Allowances for register & loop optimization, etc.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#108

Post by gavinray »

zX41ZdbW wrote: post #43

It is also worth looking at the number of unique repositories for each agent, or the number of unique large repositories (e.g., by the threshold on the number of stars). Here is the report we can check: https://play.clickhouse.com/play?user=play#V0lUSCByZXBvX3N0Y... I've also added some less popular agents like jetbrains-junie, and added a link to a random pull request for each agent, so we can look at the example PR…

This is really cool and ought to be higher up I think, especially since you can freely edit + re-run the query in the browser.

That "spark bar-chart" column output is one of the neatest things I've seen in a while. What a brilliant feature.

Re: Tracking Copilot vs. Codex vs. Cursor vs. Devin PR Performance

#110

Post by taneq »

tough wrote: post #100
taneq wrote: post #98

Earlier quoted context omitted.

Hmm, so basically automated clean room reimplementation, using coding agents? Our concepts of authorship, copying, and equivalence are getting a real workout these days!

you'd need a pretty good opsec and non-search capable agent and logs of all its actions/chain of thought/process to be able to truly claim cleanroom implementation tho

The logs and traceability are the secret sauce here. It's one thing to have an artifact that mysteriously replicates the functionality of a well known IP-protected product without just straight up copying it. It's another thing to be able to demonstrate that said artifact was generated solely from information in the public domain or otherwise legally valid to use.
Post reply on HN