Do we know how much extra work it created for the real people who had to review the proposed fixes?
Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
41–50 of 150 posts
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#4212% fix rate = 88% bug rate
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#43A 1/8 chance of fixing a bug at the cost of a careful review and some corrections is not bad. 0% -> 12% improvement is not bad for two years either (I'm somewhat arbitrary picking the release date of ChatGPT). If this can be kept up for a few years we will have some extremely useful tooling. The cost can be relatively high as well, since engineering time is currently orders of magnitude more expensive than these tool…
I still don't know. I feel like there are many ways where GPT will write some code or fix a bug in a way that makes it significantly harder to debug. Even for relatively simple tasks, it's kind of like machine-generated code that I would not want to touch.
It is a bit worrisome but we manage to deal with subpar human code as well. Often the boilerplate generated by ChatGPT is already better than what an unexperienced coder would string together. I‘m sure it will not be a free lunch but the the benefits will probably outweigh the downsides.
Interesting scalability questions will arise wrt to security when scaling the already unmanagably large code bases by another magnitude (or two), though.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#44Earlier quoted context omitted.
Only problem is that the critical `chars` method doesn't actually exist. Rust's standard library has a `chars` method for strings, but not for Readers. (Also, the comment about the iterator element type is inconsistent with the code following it. Based on the comment, `c` would be of type `(char, usize)`, but then trying to print it with {} would fail because tuples don't implement Display.)
good catch. feeding it the error output of rustc it then produces: use std::fs::File; use std::io::{self, Read}; fn read_file_character_by_character(path: &str) -> io::Result { let mut file = File::open(path)?; let mut contents = String::new(); file.read_to_string(&mut contents)?; for c in contents.chars() { println!("{}", c); } Ok(()) } fn main() { let path = "path/to/your/file.txt"; if let Err(e) = read_file_charac…
On errors, it exits with success.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#45For anyone who didn't bother looking deeper, the SWEbench benchmark contains only Python code projects, so it is not representative of all the programing languages and frameworks.
I'm working on a more general SWE task eval framework in JS for arbitrary language and framework now (for starter JS/TS, SQL and Python), for my own prompt engineering product.
Hit me up if you are interested.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#46Earlier quoted context omitted.
it does an ok job with this task: use std::fs::File; use std::io::{self, BufReader, Read}; fn read_file_character_by_character(path: &str) -> io::Result { // Open the file in read-only mode. let file = File::open(path)?; // Create a buffered reader to read the file more efficiently. let reader = BufReader::new(file); // `chars` method returns an iterator over the characters of the input. // Note that it returns a Res…
If we're being sticklers, this isn't reading character-by-character: it's performing a buffered read, which then gets iterated over.
The original prompt is a bit under-specified. (But hey, that certainly matches the real world!)
You're going to have to buffer at least a little, to figure out where the USV / grapheme boundary is, depending on our definition of "character". To me, a BufReader is appropriate here; it avoids lots of tiny reads to the kernel, which is probably the right behavior in a real case.
To me, "read character by character" vaguely implies something that's going to yield a stream of characters. (Again, for some definition there.)
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#47Interesting idea to provide the Agent-Computer Interface for it to scroll and such, interact easier from its perspective
Similar to how early computers didn't have enough ram to display the whole text file, so old programmers had to work with parts of the file at a time. It's not a bad way to get around the context window problem, which is kind of similar.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#48Very cool project! I've experimented in this direction previously, but found agentic behavior is often chaotic and leads to long expensive sessions that go down a wrong rabbit hole and ultimately fail. It's great that you succeed on 12% of swe-bench, but what happens the other 88% of the time? Is it useless wasted work and token costs? Or does it make useful progress that can be salvaged? Also, I think swe-bench is f…
Personally, I'd just use one of my local MacBook models (e.g. Mixtral 8x7b) and forget about any wasted branches & cents. My debugging time costs many orders of magnitude more than SWE-agent, so even a 5% backlog savings would be spectacular!
I’ve tried this with another similar system. FOSS LLMs including Mixtral are currently too weak to handle something like this. For me they run out of steam after only a few turns and start going in circles unproductively
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#49Their demo is so similar to the Devin one I had to go look up the Devin one to check I wasnt watching the same demo. I feel like there might be a reason they both picked Sympy. Also I rarely put weight into demos. They are usually cherry-picked at best and outright fabricated at worst. I want to hear what 3rd parties have to say after trying these things.
Maybe that's the point of this research. Hey look, we reproduced the way to game the stats a bit. I really can't tell anymore.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#50Friendly suggestion to the authors: success rates aren't meaningful to all but a handful of researchers. They should add a few examples of tests SWE-agent passed and did not pass to the README.
Unless you weren't actually that successful but need to publish a "successful" result