Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

github.com

Post reply on HN
Print view View original

101–110 of 150 posts

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#101

Post by bee_rider »

dimal wrote: post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

Maybe it just needs another, independent tool. One that detects poorly written bug reports and rejects them.

A cool thing about LLM is they have infinite patience. They can go back and forth with the user until they either sort out how to make a useable bug report, or give up.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#102

Post by bfdm »

bee_rider wrote: post #101
dimal wrote: post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

Maybe it just needs another, independent tool. One that detects poorly written bug reports and rejects them. A cool thing about LLM is they have infinite patience. They can go back and forth with the user until they either sort out how to make a useable bug report, or give up.

While it might tickle metrics the right way, frustrating a user into giving up because your bot was not satisfied is not solving their problem.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#103

Post by danenania »

barfbagginus wrote: post #90
danenania wrote: post #52

I'm working on a somewhat similar project: https://github.com/plandex-ai/plandex While the overall goal is to build arbitrarily large, complex features and projects that are too much for ChatGPT or IDE-based tools, another aspect that I've put a lot of focus on is how to handle mistakes and corrections when the model starts going off the rails. Changes are accumulated in a protected sandbox separate from your project…

How open are you to moving plandex cloud over to AGPL? I know, tough ask right out the gate! Think about that one for a bit. How is your market testing going? Do you have contracts with clients amenable to let you write case studies? Do you need help selling, designing, or fulfilling these kinds of pilot contacts? What are your plans for docs a PR? As a researcher, it's currently hard to situate plandex against exist…

It’s AGPL licensed already :)

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#104

Post by aiauthoritydev2 »

drcode wrote: post #61
dimal wrote: post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

> Most bug reports you get in the wild are more along the lines of Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.

12% is a very very large number for that kind of problem. I doubt even 0.1% of bug reports in the wild are that well written.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#106

Post by JonChesterfield »

If AI generated pull requests become a popular thing we'll see the end of public bug trackers.

(not because bugs will be gone - because the cost of reviewing the PR vs the benefit gained to the project will be a substantial net loss)

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#107

Post by labster »

Havoc wrote: post #77
ActionHank wrote: post #68

Earlier quoted context omitted.

Jokes on you, let me tell you about my prompt to binary LLM project. Hello world is 10GB, but even grandma can make hello worlds now.

But does it contain a heavily obfuscated back door?

Why does it take so long to get changes to your LLM merged? This is ridiculous. Please appoint Havoc as a maintainer already.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#108

Post by stingraycharles »

dimal wrote: post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

It appears that they’re using the PRs from the top5000 most popular PyPi packages for their bench: https://github.com/princeton-nlp/SWE-bench/tree/main/swebenc...

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#109

Post by gloosx »

sumeruchat wrote: post #83
recursive wrote: post #60

Earlier quoted context omitted.

Shipping like crazy isn't useful by itself. Shipping non-garbage and being able to maintain it still has some value.

Would you say cloning a complex saas startup in a week with payments integrated after letting AI just scrape them (or uploading screenshots of their app) is creating value?

Before you sold it to anyone it will only create bills. Development is such a minuscule part of a successful startup

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#110

Post by throwup238 »

bfdm wrote: post #102
bee_rider wrote: post #101

Earlier quoted context omitted.

Maybe it just needs another, independent tool. One that detects poorly written bug reports and rejects them. A cool thing about LLM is they have infinite patience. They can go back and forth with the user until they either sort out how to make a useable bug report, or give up.

While it might tickle metrics the right way, frustrating a user into giving up because your bot was not satisfied is not solving their problem.

I think that depends on the exact KPI.
Post reply on HN