Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

github.com

Post reply on HN
Print view View original

111–120 of 150 posts

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#111

Post by 1letterunixname »

scottlamb wrote: post #88
1letterunixname wrote: post #76

Earlier quoted context omitted.

There most certainly is getwchar() and fgetwc()/getwc() on anything that's POSIX C95, so that's more or less everything that's not a vintage antique. Reading individual UTF-8 codepoints is a trivial exercise if byte width getchar() were available, and portable C code to do so would be able to run on anything made after 1982. IIRC, they don't teach how to write portable C code in Comp Sci programs anymore and it's a s…

> There most certainly is getwchar() and fgetwc()/getwc() on anything that's POSIX C95, so that's more or less everything that's not a vintage antique. Apologies for the imprecision: by OS API, I meant syscall, at least on POSIX systems. The functions you refer to are C stdio things. Note also they implement on top of read(2) one of the two options I mentioned: "loop over getting the next N bytes and getting all comp…

Yes, the userland side presented such as with POSIX like ssize_t read(int fd, void* buf, size_t count). Calling that with count = 1 each time would be wasteful, but certainly libc's have been buffering this since at least the 1980's. I remember this was the case with Borland C/C++.

> Our comments are part of a thread discussing this prompt [2] that specifically requests Rust and this snippet in response [3]. Not portable C code. You can use those C stdio functions from Rust, but you really shouldn't without a very good reason. Rust has its own IO library that is safe and well integrated with other Rust things like `#![derive(Debug)]`.

Duh. It doesn't really matter what Rust has have went it comes to enabling the use of specific edge-case performance improvements for specific purposes. Inefficient AI-generated code without a clue of other approaches doesn't move the needle. Religious purity doesn't matter, only results matter.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#112

Post by itsgrimetime »

JonChesterfield wrote: post #106

If AI generated pull requests become a popular thing we'll see the end of public bug trackers. (not because bugs will be gone - because the cost of reviewing the PR vs the benefit gained to the project will be a substantial net loss)

It’ll likely keep getting better, if it gets to 30-40% I’d say that’s a decent trade off. Also could you boost your chances by having the AI do a 2nd pass and double check the work? I’d be curious what the success rate of an LLM “determining whether a bug fix is valid” is

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#113

Post by CGamesPlay »

JonChesterfield wrote: post #106

If AI generated pull requests become a popular thing we'll see the end of public bug trackers. (not because bugs will be gone - because the cost of reviewing the PR vs the benefit gained to the project will be a substantial net loss)

Not a chance. If AI-generated pull requests become popular, GitHub will automatically offer them in response to opened issues. Case in point: they already are popular for dependency upgrades.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#114

Post by senko »

anotherpaulg wrote: post #16

Very cool project! I've experimented in this direction previously, but found agentic behavior is often chaotic and leads to long expensive sessions that go down a wrong rabbit hole and ultimately fail. It's great that you succeed on 12% of swe-bench, but what happens the other 88% of the time? Is it useless wasted work and token costs? Or does it make useful progress that can be salvaged? Also, I think swe-bench is f…

If you don't mind me asking, which agentic tools/frameworks have you tried for code fixing/generation, with which LLMs?

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#115

Post by JonChesterfield »

CGamesPlay wrote: post #113
JonChesterfield wrote: post #106

If AI generated pull requests become a popular thing we'll see the end of public bug trackers. (not because bugs will be gone - because the cost of reviewing the PR vs the benefit gained to the project will be a substantial net loss)

Not a chance. If AI-generated pull requests become popular, GitHub will automatically offer them in response to opened issues. Case in point: they already are popular for dependency upgrades.

And thus issues will no longer be opened

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#116

Post by citrin_ru »

hvis wrote: post #73
dimal wrote: post #69

Earlier quoted context omitted.

I suppose I should nail down my point. No one would ever write a big report like this. A bug generally has an unknown cause. Once you found the cause of the bug, you’d fix it. Nowadays, you could just cut and paste the problem into ChatGPT and get the answer right then. So why would anyone ever log this bug? All this demo proves that they automated a process that didn’t need automation.

To be fair, sometimes meticulous users investigate the bugs and write down logical chains explaining the causes and even offer a solution at the end (which they can't apply for the lack of commit access, for instance). The proposed solution isn't always right, of course, but it would be incorrect to say that no bug reports come with a diagnosed cause. But that's exactly where a conscious reviewer is most needed, I be…

I sometimes write a detailed bug reports but not a PR when there are different ways to address the problem (and all look bad to me) or the fix can introduce new problems. But I would expect LLM to ignore tradeoffs and choose an option which not necessarily the best for the same reason I hesitate - luck of understanding of this specific project.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#118

Post by Madmallard »

What veterans in the field know that AI hasn’t tackled is that the majority of difficulty in development is dealing with complexity and ambiguity and a lot of it has to do with communication between people in natural language as well as reasoning in natural language about your system. These things are not solved by AI as it is now. If you can fully specify what you want with all of the detail and corner cases and situation handling then at some point AI might be able to make all of that for you. Great! Unfortunately, that’s the actual hard part! Not the implementation generally.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#119

Post by forty »

dimal wrote: post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

The trick is that people would use LLM to write very long and detailed bug reports :p

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#120

Post by littlestymaar »

aiauthoritydev2 wrote: post #104
drcode wrote: post #61

Earlier quoted context omitted.

> Most bug reports you get in the wild are more along the lines of Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.

12% is a very very large number for that kind of problem. I doubt even 0.1% of bug reports in the wild are that well written.

Except this is automated, so you could get multiples orders of magnitude more bug filled, so you need to have a very low false positive ratio to avoid being overwhelmed by automatically generated crap (which is basically spam).
Post reply on HN