Eventually it will be 90% fix rate and everyone cheering for the 12% will be flipping burgers instead.
Flipping burgers will be automated long before AI fixes any relevant number of bug reports.
Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
121–130 of 150 posts
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#122Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#123Earlier quoted context omitted.
While it might tickle metrics the right way, frustrating a user into giving up because your bot was not satisfied is not solving their problem.
I think that depends on the exact KPI.
KPI stands for key performance indicator. It is a tool to grade people or teams by applying numbers to their work.
The only relationship you can have between these is that a ticket with a "resolved" status can be used as a KPI, but you're trying to invert the relationship here, which doesn't work. After all, it's an indicator and not a causal relationship
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#124Earlier quoted context omitted.
> Most bug reports you get in the wild are more along the lines of Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.
12% is a very very large number for that kind of problem. I doubt even 0.1% of bug reports in the wild are that well written.
Have the LLM rewrite the bug reports.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#125A 1/8 chance of fixing a bug at the cost of a careful review and some corrections is not bad. 0% -> 12% improvement is not bad for two years either (I'm somewhat arbitrary picking the release date of ChatGPT). If this can be kept up for a few years we will have some extremely useful tooling. The cost can be relatively high as well, since engineering time is currently orders of magnitude more expensive than these tool…
If someone submitted 8 PRs and 7 of them were bullshit, I would close anything else they submitted in the future without even bothering to review.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#126The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…
Maybe it would be better if the agent would help people submit better reports instead of trying to fix it. E.g. it could ask them to add missing information, test different combinations of inputs, etc. I could also learn which maintainer to ping according to the type of issue.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#127I think that "Demo" link is just an extremely annoying version of an HTML presentation, so they could save me a shitload of clicking if they just dumped their presentation out to a PDF or whatever so I could read faster than watching it type out text as if it was live. It also whines a lot on the console about its inability to connect to a websocket server on 3000 but I don't know what it would do with a websocket co…
Probably created with an LLM.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#128Earlier quoted context omitted.
Hypothetically, which ticker symbols would you buy put contracts on, at what strike prices, and at what expiration dates? As far as I can tell, a lot of people are betting a lot of money that you are wrong, but actually I think you are right.
Ugh, I am not claiming that LLMs are not great innovation. Just that they are not going to replace SWE jobs in our(maybe my) lifetime.
Conservatively, I think LLMs will replace SWE roles within the next 5-10 years. In that period SWE roles will change drastically, to be more hearding of AI agents.
We can't hope to compete with something that can edit all the files in less than 5 minutes.
If you're not at least trying to plan what your life will look like under these circumstances then you're doing yourself a disservice.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#129If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…
The way I see it, its undetermined if Generative AI will be able to fully do a SWE job. But, for most of the debates I've seen, I don't think it the answer matters all too much. Once we have models that can act as full senior SWEs.. the models can engineer the models. And then we've hit the recursive case. Once models can engineer models better and faster than humans, all bets are off. Its the foggy future. Its the s…
People (SWEs) don't want to hear this. I think it's an inevitability that something of this nature will happen.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#130Earlier quoted context omitted.
Would you say cloning a complex saas startup in a week with payments integrated after letting AI just scrape them (or uploading screenshots of their app) is creating value?
Before you sold it to anyone it will only create bills. Development is such a minuscule part of a successful startup
On vercel its free to deploy a complex app almost.