But can their AI quietly introduce a security exploit into a GitHub project?
Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
91–100 of 150 posts
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#92The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…
> Most bug reports you get in the wild are more along the lines of Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.
It fixes 12% of their benchmark suite, not 12% of bug reports.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#93I would like something like this that helps me, as a green developer, find open source projects to contribute to.
For instance, I recently learned about how to replace setup.py with pyproject.toml for a large number of projects. I also learned how to publish packages to pypi. These changes significantly improve project ease and accessibility, and are very easy to do.
The main thing that holds people back is that python packaging documentation is notoriously cryptic - well I've already paid that cost, and now it's easy!
So I'm thinking of finding projects that are healthy, but haven't focused on modernizing their packaging or distributing their project through pypi.
I'd build human + agent based tooling to help me find candidates, propose the improvement to existing maintainers, then implement and deliver.
I could maybe upgrade 100 projects, then write up the adventure.
Anyone have inspiration/similar ideas, and wanna brainstorm?
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#9412% fix rate = 88% bug rate
Yep. After xz we don't need a bot mindlessly fixing all suggestions from malicious actors
I don't think xz makes a difference here. The perceived likelihood of problems, malicious or not, is pretty much the same. As far as this discussion goes, it's just another example in the pile of examples, not an event with meaningful before and after epochs.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#95The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…
Exactly. This is not perfect and doesn't fix every report so it is useless.
On the contrary, it’s worse than useless. If it could fix 12% of bugs (it can’t — it only fixes 12% of their benchmark suite), you’d still have to figure out which 12% of the responses it gave were good. So, 88% of the time you’d have wasted time confirming a “fix” that doesn’t work. But it’s worse than that. Because even on the fixes it got right, you’d still have to fully vet it, because this tool doesn’t know when it can’t solve something, or ask for clarification. It just gives a wrong answer.
So you didn’t save 12% of your effort, you wasted probably more than double your effort checking the work of a tool that is wrong eight out of nine times.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#96I would like something like this that helps me, as a green developer, find open source projects to contribute to. For instance, I recently learned about how to replace setup.py with pyproject.toml for a large number of projects. I also learned how to publish packages to pypi. These changes significantly improve project ease and accessibility, and are very easy to do. The main thing that holds people back is that pyth…
...or you could just use the GitHub API to find projects that match certain criteria (e.g., no pyproject.toml). I'm not sure what the stochastic parrot adds here, besides making noob mistakes that you'll have to find and fix before you can submit PRs. You'd learn a lot more by trying to actually automate the process yourself.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#97The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…
I agree that bugs aren't as well specified as the example. But a specification for a new feature certainly can be.
I'm going to give it a try on my side project and see if it can at least provide a hint or some guidance on the development of small new features in an existing well structured project.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#98The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…
> Most bug reports you get in the wild are more along the lines of Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.
In my 15 years i would say less than 1% of bug reports are like this. If you know the bug to this level most people just would fix it themselves
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#99But can their AI quietly introduce a security exploit into a GitHub project?
Copilot already does this.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#100Earlier quoted context omitted.
Shipping like crazy isn't useful by itself. Shipping non-garbage and being able to maintain it still has some value.
Would you say cloning a complex saas startup in a week with payments integrated after letting AI just scrape them (or uploading screenshots of their app) is creating value?
Not without more information.