Earlier quoted context omitted.
Before you sold it to anyone it will only create bills. Development is such a minuscule part of a successful startup
On vercel its free to deploy a complex app almost.
free cheese is only in a mousetrap ;)
Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
141–150 of 150 posts
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#142Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#143Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#144The demo shows a very clearly written bug report about a matrix operation that’s producing an unexpected output. Umm… no. Most bug reports you get in the wild are more along the lines of “I clicked on on X and Y happened” then if you’re lucky they’ll say “and I expected Z”. Usually the Z expectation is left for the reader to fill in because as human users we understand the expectations. The difficulty in fixing a bug…
Exactly. This is not perfect and doesn't fix every report so it is useless.
That’s not what I said and you know it. I’m not saying LLMs are useless. I’m not even saying this tool is useless. I’m saying I’m not impressed with this tool, at least as represented in the demo.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#145Eventually it will be 90% fix rate and everyone cheering for the 12% will be flipping burgers instead.
Why would a human ever flip a burger at that point? It's not a particularly difficult task for a robot.
Unclogging sewers on the other hand…
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#146Earlier quoted context omitted.
How open are you to moving plandex cloud over to AGPL? I know, tough ask right out the gate! Think about that one for a bit. How is your market testing going? Do you have contracts with clients amenable to let you write case studies? Do you need help selling, designing, or fulfilling these kinds of pilot contacts? What are your plans for docs a PR? As a researcher, it's currently hard to situate plandex against exist…
It’s AGPL licensed already :)
Did I miss the plandex-cloud repo? It seems like it's proprietary at this time. I couldn't find the AWS design, billing system, user dashboards, and admin dashboards.
Can you point me to the missing code?
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#147Earlier quoted context omitted.
Depends on how many security vulnerabilities are in that payments system. Or, I suppose, depending on whose value. The consultants that'll have to be hired by the poor shmuck who paid for that will make a fortune auditing and cleaning up the code.
None because 1) this is pretty standard stuff with stripe 2) the good developer can go through the code and fix them in a few hours if there were any
AI will quite readily write bad quality code with security vulnerabilities even for bog standard stuff (like say SQL injections).
And sure, a good developer can fix it if they will see it. But they won't when running on that kind of schedule.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#148Earlier quoted context omitted.
I'm not afraid of LLMs replacing me because of their output quality. The problem is the proliferation of quantity-over-quality "churn out barely-working crap as fast as possible" culture that gives LLMs the advantage over real humans.
I'm kinda hoping that LLMs will get pushed into production use writing code before they have acceptable quality (because greed), and the result will be lots of crap that's so badly broken most of the time that there will be a massive pushback against said culture from the users. Maybe from the governments as well, after a few well-publicized infrastructure failures.
Unfortunately, "lots of crap that's so badly broken most of the time" already describes a lot of software these days, and yet there hasn't been much pushback. Everyone seems to be mostly in a state of learned helplessness.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#149Earlier quoted context omitted.
12% is a very very large number for that kind of problem. I doubt even 0.1% of bug reports in the wild are that well written.
Have the LLM rewrite the bug reports.
You'd want three LLMs, one to create the bugs, one to report it, one to fix it. I joke of course but on the other hand this is potentially a worthwhile architecture from a self-training perspective - a bug-creating LLM means your training set size is as big as you want it +/- GAN features.
Re: Princeton group open sources "SWE-agent", with 12% fix rate for GitHub issues
#150Earlier quoted context omitted.
I'm kinda hoping that LLMs will get pushed into production use writing code before they have acceptable quality (because greed), and the result will be lots of crap that's so badly broken most of the time that there will be a massive pushback against said culture from the users. Maybe from the governments as well, after a few well-publicized infrastructure failures.
Unfortunately, "lots of crap that's so badly broken most of the time" already describes a lot of software these days, and yet there hasn't been much pushback. Everyone seems to be mostly in a state of learned helplessness.
There is pushback, it's just not broad enough yet. Because, as broken as things are (which is especially visible to those of us making the sausage or watching it made), they still kinda sorta work most of the time, to the point where users grumble but learn to live with it.
But I think that AI coding will upset this equilibrium by reducing the quality even more, and significantly enough that users will very much notice - and for many of them it will push things into "what I need doesn't work most of the time" category. And then there will be payback.
Then again, I am an optimist.