Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Devin: AI Software Engineer

cognition-labs.com

Post reply on HN
Print view View original

251–260 of 604 posts

Re: Devin: AI Software Engineer

#251

Post by singularity2001 »

Interesting: The last demo on the blog took 2.5h to complete: https://www.cognition-labs.com/blog https://www.youtube.com/watch?v=UTS2Hz96HYQ "Devin's Upwork Side Hustle"

I wonder how much time of this was consumed by manually directing Devin into the right direction, manually fixing and undoing the mess Devin produced and watching Devin burn through $$$. As others said, being completely non-transparent about this burns a bit of trust, but I'd really like to know where we are right now. Since Devin is currently "invite only demos", a more realistic peek into the state of the art can be seen here: https://docs.sweep.dev/blogs/gpt-4-modification

My gut feeling (and limited experience): gpt-4 and other models are not quite there yet, but whoever prepares for the next generation of models now will eventually win big times. Or be replaced by simpler approaches.

Re: Devin: AI Software Engineer

#252

Post by m3kw9 »

Until you can point out via video what is the issue(“see? Here it flickers a bit and here needs centering” or when you talk to the “swe agent” and say we need this feature taken out for now, and later you ask it to put the feature back in and it remembers it had code implemented at GitHub commit id xxyyzz, you really can’t call this a software engineer

Re: Devin: AI Software Engineer

#253

Post by pstorm »

Oras wrote: post #238

From their twitter: > When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. While it is a progress, its far away from being useful to be a software engineer.

What percent does an average junior engineer solve? If it is even close, these models can be run all day and night for cheaper than one yearly SWE salary.

Re: Devin: AI Software Engineer

#254

Post by Rumudiez »

golergka wrote: post #167
MSFT_Edging wrote: post #67

Humans seek work that provides satisfaction and meaning in their life. For every technological advancement, artisans are the first to be made obsolete. Sure we have landfills full of unworn textiles, the market says its good, but overall, we keep destroying what allows humans to seek meaning. Our governments and society have made it clear, if you don't produce value, you don't deserve dignity. We have outsourced art…

> What will be left? Where will we be? Just stranded without dignity or purpose, left to rot when we no longer produce value. Nobody stops you from paying $1000 for a shirt made by artisans right now. Do you want to? > Our governments and society have made it clear, if you don't produce value, you don't deserve dignity. It's not somebody else who decided that. It's you.

I have a few handmade shirts that were all between $70-100. I buy the occasional oddity from Etsy and those things (the ones I buy) are all handmade and most of them are quite affordable – on par with shopping at Target for comparison. I’m 100% certain the artists are thankful they can be professional creators instead of becoming a wage worker or living off of some form of basic income

Re: Devin: AI Software Engineer

#255

Post by beepbooptheory »

Eager wrote: post #218
parhamn wrote: post #148

Earlier quoted context omitted.

You really need to try Opus. Try a provider that works across models (one in my bio).

It's incredible how far behind HN of all places is w.r.t. what the current best tech is. So many people talking about GPT-4 here, or even 3.5 when the SOTA has moved way along. Gemini Advanced is also a great model, but for other reasons. That thing really knows a boat load of low level optimization tricks.

I'm sure you know what your talking about, but pushing the point that what is "best" or worth talking about is something that changes like every month does not really help defend against the case that most of this is just hype-churn or marketing.

Re: Devin: AI Software Engineer

#256

Post by [deleted] »

MSFT_Edging wrote: post #67

Humans seek work that provides satisfaction and meaning in their life. For every technological advancement, artisans are the first to be made obsolete. Sure we have landfills full of unworn textiles, the market says its good, but overall, we keep destroying what allows humans to seek meaning. Our governments and society have made it clear, if you don't produce value, you don't deserve dignity. We have outsourced art…

[deleted]

Re: Devin: AI Software Engineer

#257

Post by idontpost »

mattlondon wrote: post #8

There is no way this is going to make it so that "engineers can focus on more interesting problems and engineering teams can strive for more ambitious goals." Instead it will mean that bosses can fire 75-90% of the (very expensive) engineers, with the ones who remain left to prompt the AI and clean up any mistakes/misunderstandings. I guess this is the future. We've coded ourselves out of a job. People are smiling an…

[dead]

Re: Devin: AI Software Engineer

#258

Post by 2devnull »

epolanski wrote: post #227
ThalesX wrote: post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

I'll give you examples of how it helps me: 1) copilot is a terrific auto complete, and writes tremendous amounts of repetitive boilerplate 2) copilot can help me kickstart writing some complex functions starting from a comment where I tell it what is the input and expected output. Is the implementation always perfect or bug free? No. But in general I just need to review and check rather than come up with the instruct…

But people read much less of what you type now.

Re: Devin: AI Software Engineer

#259

Post by mpalmer »

Oras wrote: post #238

From their twitter: > When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. While it is a progress, its far away from being useful to be a software engineer.

Unless humans themselves were tested on a benchmark, the benchmark data doesn't help us compare the model to human performance.

Of all the SWEs out there that draw a salary, how many do you think would improve on this 14% unassisted figure?

Re: Devin: AI Software Engineer

#260

Post by swatcoder »

pstorm wrote: post #253
Oras wrote: post #238

From their twitter: > When evaluated on the SWE-Bench benchmark, which asks an AI to resolve GitHub issues found in real-world open-source projects, Devin correctly resolves 13.86% of the issues unassisted, far exceeding the previous state-of-the-art model performance of 1.96% unassisted and 4.80% assisted. While it is a progress, its far away from being useful to be a software engineer.

What percent does an average junior engineer solve? If it is even close, these models can be run all day and night for cheaper than one yearly SWE salary.

Juniors already have bad and sometimes even negative ROI but today's working junior is the trusted engineer of tomorrow and the senior of the day after that. The problems they work on impart the knowledge and instincts that advance them through towards mastery and real value.

Budget-myopic executives already tried transfering that work to cheaper labor markets, but it worked much less than they expected and most ended up with unmaintainable software and loss of any hope for an actual engineering advantage against competitors. There's nothing new here.

There will be organizations that find a good and smart use for fully automated code generation, just like there is for outsourcing/offshoring, but it's not a universal win to just go with what's "cheaper" and organizations that don't look at the big picture are (as usual) trading short-term accounting gains for long-term value erosion.

Post reply on HN