Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

github.com

Post reply on HN
Print view View original

61–70 of 150 posts

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#61

Post by drcode »

dimal wrote: post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

> Most bug reports you get in the wild are more along the lines of

Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#62

Post by swatcoder »

a_wild_dandan wrote: post #24
anotherpaulg wrote: post #16

Very cool project! I've experimented in this direction previously, but found agentic behavior is often chaotic and leads to long expensive sessions that go down a wrong rabbit hole and ultimately fail. It's great that you succeed on 12% of swe-bench, but what happens the other 88% of the time? Is it useless wasted work and token costs? Or does it make useful progress that can be salvaged? Also, I think swe-bench is f…

Personally, I'd just use one of my local MacBook models (e.g. Mixtral 8x7b) and forget about any wasted branches & cents. My debugging time costs many orders of magnitude more than SWE-agent, so even a 5% backlog savings would be spectacular!

> My debugging time costs many orders of magnitude more than SWE-agent

Unless your job is primarily to clean up somebody else's mess, your debugging time is a key part of a career-long feedback loop that improves your craft. Be careful not to shrug it off as something less. Many many people are spending a lot of money to let you forget it, and once you do, you'll be right there in the ranks of the cheaply replaceble.

(And on the odd chance that cleaning up other people's mess is your job, you should probably be the one doing it; and for largely the same reasons)

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#63

Post by Aperocky »

a_wild_dandan wrote: post #24
anotherpaulg wrote: post #16

Very cool project! I've experimented in this direction previously, but found agentic behavior is often chaotic and leads to long expensive sessions that go down a wrong rabbit hole and ultimately fail. It's great that you succeed on 12% of swe-bench, but what happens the other 88% of the time? Is it useless wasted work and token costs? Or does it make useful progress that can be salvaged? Also, I think swe-bench is f…

Personally, I'd just use one of my local MacBook models (e.g. Mixtral 8x7b) and forget about any wasted branches & cents. My debugging time costs many orders of magnitude more than SWE-agent, so even a 5% backlog savings would be spectacular!

That's assuming that the other 95% stays the same with this new agent (vs creating more work for you to now also have to parse what the model is saying).

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#64

Post by hazn »

toddmorey wrote: post #18
matthewaveryusa wrote: post #5

Very neat. Uses the langchain method, here are some of the prompts: https://github.com/princeton-nlp/SWE-agent/blob/main/config/...

I’m always fascinated to read the system prompts & I always wonder what sort of gains can be made optimizing them further. Once I’m back on desktop I want to look at the gut history of this file.

DSPy is the best tool for optimizing prompts [0]: https://github.com/stanfordnlp/dspy

Think of it as a meta-prompt optimizer, it uses a LLM to optimize your prompts, to optimize your LLM.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#65

Post by raggi »

noncoml wrote: post #32
woodruffw wrote: post #15

Earlier quoted context omitted.

If we're being sticklers, this isn't reading character-by-character: it's performing a buffered read, which then gets iterated over.

I wouldn't say it's a nit. The file may be 10s of GB. Do you want to read it to a string?

The buffered read didn’t do that, it used the default buffered reader implementation. IIRC that implementation currently defaults to 8kb buffer windows which is a little too small to be efficient enough for high throughput, but substantially more performant than making a syscall per byte, and without spending too much memory.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#66

Post by nickpsecurity »

swatcoder wrote: post #62
a_wild_dandan wrote: post #24

Earlier quoted context omitted.

Personally, I'd just use one of my local MacBook models (e.g. Mixtral 8x7b) and forget about any wasted branches & cents. My debugging time costs many orders of magnitude more than SWE-agent, so even a 5% backlog savings would be spectacular!

> My debugging time costs many orders of magnitude more than SWE-agent Unless your job is primarily to clean up somebody else's mess, your debugging time is a key part of a career-long feedback loop that improves your craft. Be careful not to shrug it off as something less. Many many people are spending a lot of money to let you forget it, and once you do, you'll be right there in the ranks of the cheaply replaceble.…

I totally agree. My solution to this was limiting my AI use to (a) whatever didn't impair creativity and (b) just in general to keep the brain sharp. If using AI regularly, one could just manually solve a percentage of the problems.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#67

Post by toddmorey »

hazn wrote: post #64
toddmorey wrote: post #18

Earlier quoted context omitted.

I’m always fascinated to read the system prompts & I always wonder what sort of gains can be made optimizing them further. Once I’m back on desktop I want to look at the gut history of this file.

DSPy is the best tool for optimizing prompts [0]: https://github.com/stanfordnlp/dspy Think of it as a meta-prompt optimizer, it uses a LLM to optimize your prompts, to optimize your LLM.

Excellent! Thanks for sharing this!

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#68

Post by ActionHank »

r0ze-at-hn wrote: post #57
rwmj wrote: post #41

Do we know how much extra work it created for the real people who had to review the proposed fixes?

Ah, well let me tell you about my pull request reviewer LLM project.

Jokes on you, let me tell you about my prompt to binary LLM project.

Hello world is 10GB, but even grandma can make hello worlds now.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#69

Post by dimal »

drcode wrote: post #61
dimal wrote: post #55

The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…

> Most bug reports you get in the wild are more along the lines of Since this fixes 12% of the bugs, the authors of the paper probably agree with you that 100-12= 88%, and hence "most bugs" don't have nicely written bug reports.

I suppose I should nail down my point. No one would ever write a big report like this. A bug generally has an unknown cause. Once you found the cause of the bug, you’d fix it. Nowadays, you could just cut and paste the problem into ChatGPT and get the answer right then. So why would anyone ever log this bug? All this demo proves that they automated a process that didn’t need automation.

Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues

#70

Post by iwontberude »

jeremyjh wrote: post #26
iwontberude wrote: post #9

Earlier quoted context omitted.

Hypothetically, which ticker symbols would you buy put contracts on, at what strike prices, and at what expiration dates? As far as I can tell, a lot of people are betting a lot of money that you are wrong, but actually I think you are right.

The most relevant companies focused on this aren't publicly traded. The ones that are publicly traded like MSFT have way too many other factors affecting their value - not to mention the fact that they'll make money on generative AI that has nothing to do with coding regardless of if an SWE-agent ever works.

Oh well you should hear the hype from CNBC and other places, they are strongly intimating that gen AI will replace SWEs on product development teams. I totally agree it’s not likely, but it’s starting to get baked into asset prices and I want to profit from that misunderstanding.
Post reply on HN