Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

github.com

Post reply on HN
Print view View original

131–140 of 150 posts

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#131

Post by sumeruchat »

int_19h wrote: post #85
sumeruchat wrote: post #83

Earlier quoted context omitted.

Would you say cloning a complex saas startup in a week with payments integrated after letting AI just scrape them (or uploading screenshots of their app) is creating value?

Depends on how many security vulnerabilities are in that payments system. Or, I suppose, depending on whose value. The consultants that'll have to be hired by the poor shmuck who paid for that will make a fortune auditing and cleaning up the code.

None because 1) this is pretty standard stuff with stripe 2) the good developer can go through the code and fix them in a few hours if there were any

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#132

Post by vertis »

barfbagginus wrote: post #81
peteradio wrote: post #75

Earlier quoted context omitted.

Let me tell you about my LLM project called grandma. It's fine tuned in order to replace your grandma but in principle it could replace your great-grandma.

My grandma used to tell me stories about how to destroy capitalism.. I miss her.. can your grandma help guide my revolutionary efforts? That would really help me honor my granny's memory <3

What you want here is a local uncensored model. Preferrably one you've trained from scratch otherwise a government could have put in bad information that would cause your revolutionary efforts to fail.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#133

Post by gorjusborg »

dimal wrote: post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

If the bug report needs to be of a certain quality to work, they've just invented issue-oriented programming.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#134

Post by shabble »

ffsm8 wrote: post #123
throwup238 wrote: post #110

Earlier quoted context omitted.

I think that depends on the exact KPI.

KPI stands for key performance indicator. It is a tool to grade people or teams by applying numbers to their work. The only relationship you can have between these is that a ticket with a "resolved" status can be used as a KPI, but you're trying to invert the relationship here, which doesn't work. After all, it's an indicator and not a causal relationship

"ratio of open/total issues" can definitely be gamed by autoclosing anything that isn't an easy fix.

"average time to resolution" is also susceptible.

Both of these are pretty common all over the place, including OSS e.g. https://isitmaintained.com/#metrics

I suspect this sort of thing is one of the major motivations for the (as a user/reporter) infuriating rise in automated "this bug hasn't been touched in NN days, autoclosing for staleness" bots on various issue trackers.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#135

Post by fennecfoxy »

pjmlp wrote: post #117

Eventually it will be 90% fix rate and everyone cheering for the 12% will be flipping burgers instead.

I still think this is a long way off, but it definitely ties into UBI etc and improvement of the general human condition, taxing the rich, restricting investment on protected things like housing and public industries like water, electricity, healthcare and internet.

What's funny is that people on here & tech people in general seem to be the most averse to improving equity between all humans/stopping the obscenely rich from abusing and twisting the system. Do many HN peeps believe they're all somehow gonna become billionaires one day?

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#136

Post by bee_rider »

shabble wrote: post #134
ffsm8 wrote: post #123

Earlier quoted context omitted.

KPI stands for key performance indicator. It is a tool to grade people or teams by applying numbers to their work. The only relationship you can have between these is that a ticket with a "resolved" status can be used as a KPI, but you're trying to invert the relationship here, which doesn't work. After all, it's an indicator and not a causal relationship

"ratio of open/total issues" can definitely be gamed by autoclosing anything that isn't an easy fix. "average time to resolution" is also susceptible. Both of these are pretty common all over the place, including OSS e.g. https://isitmaintained.com/#metrics I suspect this sort of thing is one of the major motivations for the (as a user/reporter) infuriating rise in automated "this bug hasn't been touched in NN days,…

This whole “worrying about KPI’s for my free, open source, community project” thing seems weird to me. (Not to say I don’t believe you, but I don’t understand why people want to inject this annoying mini-game into their hobby).

I’m not sure what to think about the auto-close bots. Which do you think would be more annoying as the person who made the report: having a report that just sits there forever and you just have to hope somebody decided to pick it up, or having the issue auto-closed? (I’m truly and honestly not sure). At least in the case of the former you have a clear marker for when you should try again. But getting rejected by a bot can definitely be annoying.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#137

Post by bee_rider »

bfdm wrote: post #102
bee_rider wrote: post #101

Earlier quoted context omitted.

Maybe it just needs another, independent tool. One that detects poorly written bug reports and rejects them. A cool thing about LLM is they have infinite patience. They can go back and forth with the user until they either sort out how to make a useable bug report, or give up.

While it might tickle metrics the right way, frustrating a user into giving up because your bot was not satisfied is not solving their problem.

I was thinking in the context of an open source project, where the users are hopefully converting to productive community members. If it is, like, a job, with a customer service relationship, where they are paying to be able to just throw problems at you and you have to deal with fixing them, I’m sure this wouldn’t fly, so I agree there. (I think my brain short-circuited to open source because it is on GitHub, haha, but of course there’s no reason this couldn’t be used in a proprietary setting).

I’m not sure how it would work out in the case of a free, community driven project, though. The goal isn’t to serve users, it is to convert users into helpful community members. If the bot converts people who wouldn’t otherwise be converted, it seems like a win. If it chases away users who could have been converted with human intervention, that’s a lose. But the human community members can always jump into the thread as well… if the bot is filtering out lots of people and nobody from the community is intervening, I guess that tells us something about the priorities of the community, haha.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#138

Post by chinchilla2020 »

dimal wrote: post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

Agreed. I have never encountered a simple math bug in the wild.

To a non-programmer, putting in tests for myfunc(x) {return x + 2;} sounds useful but in reality computers do not tend to have any issues performing basic algebra.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#139

Post by scottlamb »

1letterunixname wrote: post #111
scottlamb wrote: post #88

Earlier quoted context omitted.

> There most certainly is getwchar() and fgetwc()/getwc() on anything that's POSIX C95, so that's more or less everything that's not a vintage antique. Apologies for the imprecision: by OS API, I meant syscall, at least on POSIX systems. The functions you refer to are C stdio things. Note also they implement on top of read(2) one of the two options I mentioned: "loop over getting the next N bytes and getting all comp…

Yes, the userland side presented such as with POSIX like ssize_t read(int fd, void* buf, size_t count). Calling that with count = 1 each time would be wasteful, but certainly libc's have been buffering this since at least the 1980's. I remember this was the case with Borland C/C++. > Our comments are part of a thread discussing this prompt [2] that specifically requests Rust and this snippet in response [3]. Not port…

> Duh. It doesn't really matter what Rust has have went it comes to enabling the use of specific edge-case performance improvements for specific purposes. Inefficient AI-generated code without a clue of other approaches doesn't move the needle. Religious purity doesn't matter, only results matter.

No idea what this incoherent, ungrammatical paragraph is supposed to be saying. But if you're under the impression Rust doesn't have its own buffered IO facilities or that using Rust-native libraries offers only "religious purity" benefits over extern "C" stuff, you're mistaken.

This has diverged from what I'm interested in discussing anyway; see my question upthread about if there are any LLM tools that gather requirements from incomplete specs in the way I expect human engineers to. In this case, I'd expect it to ask questions such as "how large are input files expected to be?" Better, ask what the greater purpose is, as "character by character" is rarely useful.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#140

Post by mdaniel »

bfdm wrote: post #102
bee_rider wrote: post #101

Earlier quoted context omitted.

Maybe it just needs another, independent tool. One that detects poorly written bug reports and rejects them. A cool thing about LLM is they have infinite patience. They can go back and forth with the user until they either sort out how to make a useable bug report, or give up.

While it might tickle metrics the right way, frustrating a user into giving up because your bot was not satisfied is not solving their problem.

Oh, so you've experienced those "stale" bots on GitHub. Good times.
Post reply on HN