I recommend looking at swe-bench to get an idea as to what breakthroughs this product accomplishes: https://www.swebench.com/. They claim to have tested SOTA models like GPT-4 and Claude 2 (I would like to see it tested on Claude 3 Opus) and their score is 13.86% as opposed to 4.80% for Claude 2. This benchmark is for solving real-world GitHub issues. So for those claiming that they tried models in the past and it didn't work for their use case, maybe this one will be better?
Devin: AI Software Engineer
241–250 of 604 posts
Re: Devin: AI Software Engineer
#242Earlier quoted context omitted.
What kind of work do you have in mind?
In terms of my "off ramp"? I have a multi-part plan. Immediately, i'm working to get closer to the business. To be closer to the position of defining requirements, not implementing requirements. Secondary, i'm experimenting with ideas I hope can become a business. as a final fallback, I have a hobby woodshop in my basement, and I love making furniture.
Making furniture! but we've had machines to do that for 100 years!
Programmers will be fine. The AI plagiarism engines are severely limited, and will be for the forseeable future. Maybe someday i'll be equivalent to the ren-faire blacksmith, but I'm gonna do this until I die.
Re: Devin: AI Software Engineer
#243As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…
I'll give you examples of how it helps me: 1) copilot is a terrific auto complete, and writes tremendous amounts of repetitive boilerplate 2) copilot can help me kickstart writing some complex functions starting from a comment where I tell it what is the input and expected output. Is the implementation always perfect or bug free? No. But in general I just need to review and check rather than come up with the instruct…
I agree with all of your points and experience the same benefits.
1) Autocomplete is more often than not what I want or pretty darn close.
2) Sometimes I need a discrete function that I am not sure how I want to write. I use a prompt with 3.5/4 inside of my IDE to ask it to write that function.
It is definitely not writing complete programs any time soon but I can see where it's heading in the near term. Couple it with something like RAG to answer questions on library/api implementations. Maybe give it a stronger opinion about what good Python code looks like.
For the naysayers I don't know how you use it but it is certainly useful enough for me to pay for.
Re: Devin: AI Software Engineer
#244As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…
I totally agree!
And I'm sure the reason for that is the garbage input. From time to time I have to perform quantitative code analyses in our so called enterprise repositories. And the results are shocking every time. I have found an extremely poor SQL code block to type cast many columns in hundreds of projects. It was simply copied again and again even though the casting was no longer necessary.
The training base should be sufficiently qualified (and StackOverflow ranking is obviously not enough).
But unfortunately it's probably too late for that now. Now inexperienced programmers are undercutting themselves with poor AI output as training input for the next generation of models.
Re: Devin: AI Software Engineer
#245Earlier quoted context omitted.
Call me fool but s this hype reminds me the hype about self-driving cars coming in a year back in 2014. AI engineers are coming, but I think we would have long retired before they reach a point where they can replace us.
The difference with self-driving cars hype is that they need to be 99.999% good so pretty much perfect to be useful on road and be incorporated mainstream. AI doing some tasks 90% as good as human is good enough. Self driving cars got massively improved in the last 15 years. I remember 15 years ago when DARPA were doing their first self driving challenge and the current tech we have is like magic comparing to what we…
But with software doesn't technical debt accumulate over time when low quality engineers keep working on it?
That's why starting projects with something like Cursor makes you seem superhuman, but as the project grows the AI is more likely to get stuck because of previous low-quality choices. Just like with driving cars, it seems like you need strong supervision. (at least for the foreseeable future)
Re: Devin: AI Software Engineer
#246Yet again, bad time to be on the labor side of the equation, great time to be a capitalist. For us laborers, if I had to choose from a list of fields to go into, anything creative would be low on the list. 'Prompt Engineer' will be the only one left. UBI is a pipe dream... it's not happening. The wealth and means of production won't be shared in any meaningful capacity. Wealth inequality can get a whole lot worse.
The measure of bullshit in this field is promoting the term 'prompt engineering'. It's prompt futzing or prompt fiddling. There's no engineering involved.
Re: Devin: AI Software Engineer
#247As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…
I'll give you examples of how it helps me: 1) copilot is a terrific auto complete, and writes tremendous amounts of repetitive boilerplate 2) copilot can help me kickstart writing some complex functions starting from a comment where I tell it what is the input and expected output. Is the implementation always perfect or bug free? No. But in general I just need to review and check rather than come up with the instruct…
I ended up getting annoyed with the autocomplete feature taking over things such as snippet expansion in vscode, so I turned it off personally. I felt that the battling against the assistant made around a break even productivity gain overall. Except for regular expressions, that I have basically offloaded to AI almost in its entirety for non trivial things.
Re: Devin: AI Software Engineer
#248As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…
As with everything about AI, HN once again shows a remarkable inability to project into the future. This site has honestly been absolutely useless when discussing new technology now. No excitement, no curiosity. Just constantly crapping on anything new and lamenting that a brand new technology is not 100% perfect within a year of launch. Remove "Hacker" from this site's name, because I see none of that spirit here an…
Are you kidding me, I'd say it's 80% people hyping up AI.
Re: Devin: AI Software Engineer
#249From their twitter: > When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. While it is a progress, its far away from being useful to be a software engineer.
13% unassisted is crazy. That's probably half the performance of an intern that costs ~100k/year.
Re: Devin: AI Software Engineer
#250Earlier quoted context omitted.
A lot of startups are selling the dream/hype of not ever having to learn to code. Be aware that it’s hype. Learn to code if you want to build stuff. They will be tools for those that have the knowledge needed to effectively use them.
Reminds me of the no-code / low-code hype around 2020, tons of startups advertising app-builders that used little, if any, AI. Just blocks that you dragged-and-dropped. While many of them were successful, it seems like overall they didn't really make much of a dent in industry, which I found very curious. Like, by now you'd think it would be inevitable that we wouldn't be writing software in a text-editor or IDE. Eve…
[dead]