Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

github.com

Post reply on HN
Print view View original

11–20 of 150 posts

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#11

Post by vineyardmike »

noncoml wrote: post #8

If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…

Yea the problem with that is the control group - grab any SWE and ask them the same thing. I don’t think most would pass. Unless you want to give an SWE time to learn… then it’s hardly fair. And I vaguely trust the LLM to be able to learn it too.

Also I just asked Claude and Gemini and they both provided an implementation that matches the “bytes to UTF-8” rust docs. Assuming those are right,LLMs can do this (but I haven’t tested the code).

https://doc.rust-lang.org/std/string/struct.String.html

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#13

Post by raggi »

noncoml wrote: post #8

If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…

it does an ok job with this task:

    use std::fs::File;
    use std::io::{self, BufReader, Read};

    fn read_file_character_by_character(path: &str) -> io::Result {
        // Open the file in read-only mode.
        let file = File::open(path)?;

        // Create a buffered reader to read the file more efficiently.
        let reader = BufReader::new(file);

        // `chars` method returns an iterator over the characters of the input.
        // Note that it returns a Result, where usize is the byte length of the char.
        for char_result in reader.chars() {
            match char_result {
                Ok(c) => print!("{}", c),
                Err(e) => return Err(e),
            }
        }

        Ok(())
    }

    fn main() {
        let path = "path/to/your/file.txt";

        if let Err(e) = read_file_character_by_character(path) {
            eprintln!("Error reading file: {}", e);
        }
    }

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#14

Post by unit_circle »

A 1/8 chance of fixing a bug at the cost of a careful review and some corrections is not bad.

0% -> 12% improvement is not bad for two years either (I'm somewhat arbitrary picking the release date of ChatGPT). If this can be kept up for a few years we will have some extremely useful tooling. The cost can be relatively high as well, since engineering time is currently orders of magnitude more expensive than these tools.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#15

Post by woodruffw »

raggi wrote: post #13
noncoml wrote: post #8

If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…

it does an ok job with this task: use std::fs::File; use std::io::{self, BufReader, Read}; fn read_file_character_by_character(path: &str) -> io::Result { // Open the file in read-only mode. let file = File::open(path)?; // Create a buffered reader to read the file more efficiently. let reader = BufReader::new(file); // `chars` method returns an iterator over the characters of the input. // Note that it returns a Res…

If we're being sticklers, this isn't reading character-by-character: it's performing a buffered read, which then gets iterated over.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#16

Post by anotherpaulg »

Very cool project!

I've experimented in this direction previously, but found agentic behavior is often chaotic and leads to long expensive sessions that go down a wrong rabbit hole and ultimately fail.

It's great that you succeed on 12% of swe-bench, but what happens the other 88% of the time? Is it useless wasted work and token costs? Or does it make useful progress that can be salvaged?

Also, I think swe-bench is from your group, right? Have you made any attempt to characterize a "skilled human upper bound" score?

I randomly sampled a dozen swe-bench tasks myself, and found that many were basically impossible for a skilled human to “solve”. Mainly because the tasks were under specified wrt to the hidden test cases that determine passing. The tests were checking implementation specific details from the repo’s PR that weren't actually stated requirements of the task.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#17

Post by raggi »

raggi wrote: post #13
noncoml wrote: post #8

If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…

it does an ok job with this task: use std::fs::File; use std::io::{self, BufReader, Read}; fn read_file_character_by_character(path: &str) -> io::Result { // Open the file in read-only mode. let file = File::open(path)?; // Create a buffered reader to read the file more efficiently. let reader = BufReader::new(file); // `chars` method returns an iterator over the characters of the input. // Note that it returns a Res…

fwiw, the benchmark that matters really has nothing to do with authoring code.

the typing of code is the easy part even though it's a part a lot of folks are somewhat addicted to.

the things which have far more value are applying value judgements to requirements, correlating and incorporating sparse and inaccurate diagnostic information into a coherent debugging strategy, and so on. there will come a time when it can assist with these too, probably first on requirements distillation, but for more complex debugging tasks that's a novel problem solving area that we've yet to see substantial movement on.

so if you want to stave off the robots coming for you, get good at debugging hard problems, and learn to make really great use of tools that accelerate the typing out of solutions to baseline product requirements.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#18

Post by toddmorey »

matthewaveryusa wrote: post #5

Very neat. Uses the langchain method, here are some of the prompts: https://github.com/princeton-nlp/SWE-agent/blob/main/config/...

I’m always fascinated to read the system prompts & I always wonder what sort of gains can be made optimizing them further.

Once I’m back on desktop I want to look at the gut history of this file.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#19

Post by raggi »

woodruffw wrote: post #15
raggi wrote: post #13

Earlier quoted context omitted.

it does an ok job with this task: use std::fs::File; use std::io::{self, BufReader, Read}; fn read_file_character_by_character(path: &str) -> io::Result { // Open the file in read-only mode. let file = File::open(path)?; // Create a buffered reader to read the file more efficiently. let reader = BufReader::new(file); // `chars` method returns an iterator over the characters of the input. // Note that it returns a Res…

If we're being sticklers, this isn't reading character-by-character: it's performing a buffered read, which then gets iterated over.

here's what it does if i feed your nit pick into it verbatim:

    use std::fs::File;
    use std::io::{self, Read};
    use std::str::from_utf8;

    fn read_file_character_by_character(path: &str) -> io::Result {
        // Open the file in read-only mode.
        let mut file = File::open(path)?;

        let mut buffer = [0; 4]; // UTF-8 character max byte length
        let mut temp_buffer = vec![];

        while let Ok(bytes_read) = file.read(&mut buffer[..1]) { // Read byte by byte
            if bytes_read == 0 { // End of file
                break;
            }

            temp_buffer.push(buffer[0]);

            if let Ok(s) = from_utf8(&temp_buffer) {
                if s.chars().count() == 1 { // Complete character read
                    print!("{}", s);
                    temp_buffer.clear(); // Clear temp buffer for next character
                }
            }
        }

        Ok(())
    }

    fn main() {
        let path = "path/to/your/file.txt";

        if let Err(e) = read_file_character_by_character(path) {
            eprintln!("Error reading file: {}", e);
        }
    }

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#20

Post by comex »

raggi wrote: post #13
noncoml wrote: post #8

If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…

it does an ok job with this task: use std::fs::File; use std::io::{self, BufReader, Read}; fn read_file_character_by_character(path: &str) -> io::Result { // Open the file in read-only mode. let file = File::open(path)?; // Create a buffered reader to read the file more efficiently. let reader = BufReader::new(file); // `chars` method returns an iterator over the characters of the input. // Note that it returns a Res…

Only problem is that the critical `chars` method doesn't actually exist. Rust's standard library has a `chars` method for strings, but not for Readers.

(Also, the comment about the iterator element type is inconsistent with the code following it. Based on the comment, `c` would be of type `(char, usize)`, but then trying to print it with {} would fail because tuples don't implement Display.)

Post reply on HN