Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

github.com

Post reply on HN
Print view View original

51–60 of 150 posts

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#52

Post by danenania »

I'm working on a somewhat similar project: https://github.com/plandex-ai/plandex

While the overall goal is to build arbitrarily large, complex features and projects that are too much for ChatGPT or IDE-based tools, another aspect that I've put a lot of focus on is how to handle mistakes and corrections when the model starts going off the rails. Changes are accumulated in a protected sandbox separate from your project files, a diff review TUI is included that allows for bad changes to be rejected, all actions are version-controlled so you can easily go backwards and try a different approach, and branches are also included for trying out multiple approaches.

I think nailing this developer-AI feedback loop is the key to getting authentic productivity gains. We shouldn't just ask how well a coding tool can pass benchmarks, but what the failure case looks like when things go wrong.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#53

Post by etheridev »

danenania wrote: post #52

I'm working on a somewhat similar project: https://github.com/plandex-ai/plandex While the overall goal is to build arbitrarily large, complex features and projects that are too much for ChatGPT or IDE-based tools, another aspect that I've put a lot of focus on is how to handle mistakes and corrections when the model starts going off the rails. Changes are accumulated in a protected sandbox separate from your project…

You need to make yourself a business analyst agent to provide the feedback! To make it real, perhaps a team of them with conflicting personalities.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#55

Post by dimal »

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations.

The difficulty in fixing a bug is in figuring out what’s causing the bug. If you know it’s caused by an incorrect operation, and we know that LLMs can fix simple defects like this, what does this prove?

Has anyone dug through the paper yet to see what the rest of the issues look like? And what the diffs look like? I suppose I’ll try when I have a sec.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#58

Post by danenania »

etheridev wrote: post #53
danenania wrote: post #52

I'm working on a somewhat similar project: https://github.com/plandex-ai/plandex While the overall goal is to build arbitrarily large, complex features and projects that are too much for ChatGPT or IDE-based tools, another aspect that I've put a lot of focus on is how to handle mistakes and corrections when the model starts going off the rails. Changes are accumulated in a protected sandbox separate from your project…

You need to make yourself a business analyst agent to provide the feedback! To make it real, perhaps a team of them with conflicting personalities.

I think we'll get there at some point, but one thing I've learned from this project is how difficult it is to stack AI interactions. Each little bit of AI-based logic that gets added tends to fail terribly at first. Only after a long period of intense testing and iteration does it become remotely usable. The more you are combining different kinds of tasks, the more difficult it gets.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#59

Post by scottlamb »

phaer wrote: post #30
raggi wrote: post #22

Earlier quoted context omitted.

good catch. feeding it the error output of rustc it then produces: use std::fs::File; use std::io::{self, Read}; fn read_file_character_by_character(path: &str) -> io::Result { let mut file = File::open(path)?; let mut contents = String::new(); file.read_to_string(&mut contents)?; for c in contents.chars() { println!("{}", c); } Ok(()) } fn main() { let path = "path/to/your/file.txt"; if let Err(e) = read_file_charac…

But this doesn't read the file char-by-char, but uses buffering to read it into a string

What would you expect? There's no OS API for "read one character", except in say ASCII where 1 byte = 1 code point = 1 character. And it'd be hideously inefficient anyway. So you either loop over getting the next N bytes and getting all complete characters so far (with some extra complexity around characters that cross chunk boundaries) or you read the whole thing into a single buffer and iterate the characters. This code does the latter. If this tool doesn't have the ability to respond by asking requirements questions, I'd consider either choice valid.

Of course, in real life, I do expect to get requirements questions back from an engineer when I assign a task. Seems more practical than anticipating everything up-front into the perfect specification/prompt. Why shouldn't I expect the same from an LLM-based tool? Are any of them set up to do that?

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#60

Post by recursive »

sumeruchat wrote: post #37

Once we have this fully automated, any good developer could have a team of 100 robo SWEs and ship like crazy. The real competition is with those devs not with the bots.

Shipping like crazy isn't useful by itself. Shipping non-garbage and being able to maintain it still has some value.
Post reply on HN