Live data from Hacker News

Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

github.com

71–80 of 150 posts

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#71
post #22
post #20

Earlier quoted context omitted.

Only problem is that the critical `chars` method doesn't actually exist. Rust's standard library has a `chars` method for strings, but not for Readers. (Also, the comment about the iterator element type is inconsistent with the code following it. Based on the comment, `c` would be of type `(char, usize)`, but then trying to print it with {} would fail because tuples don't implement Display.)

good catch. feeding it the error output of rustc it then produces: use std::fs::File; use std::io::{self, Read}; fn read_file_character_by_character(path: &str) -> io::Result { let mut file = File::open(path)?; let mut contents = String::new(); file.read_to_string(&mut contents)?; for c in contents.chars() { println!("{}", c); } Ok(()) } fn main() { let path = "path/to/your/file.txt"; if let Err(e) = read_file_charac…

A few notes:

- It should be generating `path: impl AsRef` to be properly generic.

- It's not setting a nonzero exit code on error.

- Edge case handling is a vital property for production-usable tools at scale. I'm wondering if it can yet special case situations such as creating a conditional compilation version for Linux that uses the splice syscall when the arguments are 2 file handles.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#72
post #8

If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…

I'm not afraid of LLMs replacing me because of their output quality. The problem is the proliferation of quantity-over-quality "churn out barely-working crap as fast as possible" culture that gives LLMs the advantage over real humans.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#73
post #69
post #61

Earlier quoted context omitted.

> Most bug reports you get in the wild are more along the lines of Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.

I suppose I should nail down my point. No one would ever write a big report like this. A bug generally has an unknown cause. Once you found the cause of the bug, you’d fix it. Nowadays, you could just cut and paste the problem into ChatGPT and get the answer right then. So why would anyone ever log this bug? All this demo proves that they automated a process that didn’t need automation.

To be fair, sometimes meticulous users investigate the bugs and write down logical chains explaining the causes and even offer a solution at the end (which they can't apply for the lack of commit access, for instance).

The proposed solution isn't always right, of course, but it would be incorrect to say that no bug reports come with a diagnosed cause. But that's exactly where a conscious reviewer is most needed, I believe.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#74
post #65
post #32

Earlier quoted context omitted.

I wouldn't say it's a nit. The file may be 10s of GB. Do you want to read it to a string?

The buffered read didn’t do that, it used the default buffered reader implementation. IIRC that implementation currently defaults to 8kb buffer windows which is a little too small to be efficient enough for high throughput, but substantially more performant than making a syscall per byte, and without spending too much memory.

I was talking about this:

    let mut file = File::open(path)?;
    let mut contents = String::new();
    file.read_to_string(&mut contents)?;

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#75

Earlier quoted context omitted.

Ah, well let me tell you about my pull request reviewer LLM project.

Jokes on you, let me tell you about my prompt to binary LLM project. Hello world is 10GB, but even grandma can make hello worlds now.

Let me tell you about my LLM project called grandma. It's fine tuned in order to replace your grandma but in principle it could replace your great-grandma.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#76
post #30

Earlier quoted context omitted.

But this doesn't read the file char-by-char, but uses buffering to read it into a string

What would you expect? There's no OS API for "read one character", except in say ASCII where 1 byte = 1 code point = 1 character. And it'd be hideously inefficient anyway. So you either loop over getting the next N bytes and getting all complete characters so far (with some extra complexity around characters that cross chunk boundaries) or you read the whole thing into a single buffer and iterate the characters. This…

There most certainly is getwchar() and fgetwc()/getwc() on anything that's POSIX C95, so that's more or less everything that's not a vintage antique.

Reading individual UTF-8 codepoints is a trivial exercise if byte width getchar() were available, and portable C code to do so would be able to run on anything made after 1982. IIRC, they don't teach how to write portable C code in Comp Sci programs anymore and it's a shame.

Never read a file completely into memory at once unless there is zero chance of it being a huge file because this is an obvious DoS vector and waste of resources.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#77

Earlier quoted context omitted.

Ah, well let me tell you about my pull request reviewer LLM project.

Jokes on you, let me tell you about my prompt to binary LLM project. Hello world is 10GB, but even grandma can make hello worlds now.

But does it contain a heavily obfuscated back door?

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#78
post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

Exactly. This is not perfect and doesn't fix every report so it is useless.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#80

For anyone who didn't bother looking deeper, the SWEbench benchmark contains only Python code projects, so it is not representative of all the programing languages and frameworks. I'm working on a more general SWE task eval framework in JS for arbitrary language and framework now (for starter JS/TS, SQL and Python), for my own prompt engineering product. Hit me up if you are interested.

Assuming the data set is proprietary, else please share the repo
Post reply on HN