Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
1–10 of 150 posts
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#2Friendly suggestion to the authors: success rates aren't meaningful to all but a handful of researchers. They should add a few examples of tests SWE-agent passed and did not pass to the README.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#3Friendly suggestion to the authors: success rates aren't meaningful to all but a handful of researchers. They should add a few examples of tests SWE-agent passed and did not pass to the README.
Yes please, the code quality on Devin was incredibly poor in all examples I traced down.
At least from a maintainability perspective.
I would like to see if this implementation is less destructive or at least more suitable for a red-green-refactor workflow.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#4I think that "Demo" link is just an extremely annoying version of an HTML presentation, so they could save me a shitload of clicking if they just dumped their presentation out to a PDF or whatever so I could read faster than watching it type out text as if it was live. It also whines a lot on the console about its inability to connect to a websocket server on 3000 but I don't know what it would do with a websocket connection if had it
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#5Very neat. Uses the langchain method, here are some of the prompts:
https://github.com/princeton-nlp/SWE-agent/blob/main/config/...
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#6And creates how many new ones?
This and Devin generate garbage code that will make any codebase worse.
It's a joke that 12.5% is even associated with the word "success".
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#7And creates how many new ones? This and Devin generate garbage code that will make any codebase worse. It's a joke that 12.5% is even associated with the word "success".
Do spaces and spelling fixes count?
Copilot, so far, is only good for predicting the next bit of similar patterns of code
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#8If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character
Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LLM is taking. Mind you, the most popular the issue, the better the line LLM is taking. So in other words, IMHO is a glorified Stack Overflow. Just as there are engineers that copy-paste from SO without having any idea what the code does, there will be engineers that will just copy paste from LLM. Their work will be much better than if they used SO, but I think it's still nowhere to the mark of a Senior SWE and above.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#9If you are afraid that LLMs will replace you at your job, ask an LLM to write Rust code for reading a utf8 file character by character Edit: Yes, it does write some code that is "close" enough, but in some cases it is wrong, in others it doesn't not do exactly what asked. I.e. needs supervision from someone who understands both the requirements, the code and the problems that may arise from the naive line that the LL…
Hypothetically, which ticker symbols would you buy put contracts on, at what strike prices, and at what expiration dates? As far as I can tell, a lot of people are betting a lot of money that you are wrong, but actually I think you are right.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#10Their demo is so similar to the Devin one I had to go look up the Devin one to check I wasnt watching the same demo. I feel like there might be a reason they both picked Sympy. Also I rarely put weight into demos. They are usually cherry-picked at best and outright fabricated at worst. I want to hear what 3rd parties have to say after trying these things.