Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on a month with Devin

answer.ai

Post reply on HN
Print view View original

71–80 of 194 posts

Re: Thoughts on a month with Devin

#71

Post by rco8786 »

I'm sure a lot of folks in these comments predicted these sorts of results with surprising accuracy.

Stuff like this is why I scoff when I hear about CEOs freezing engineering hiring or saying they just don't need mid-level engineers anymore because they have AI.

I'll start believing that when I see it happening, and see actual engineers saying that AI can replace a human.

I am long AI, but I think the winning formula is small, repetitive tasks with a little too much variation to make it worth it (or possible) to automate procedurally. Pulling data from Notion into Google sheets, like these folks did initially, is probably fine. Having it manage your infrastructure and app deployments, likely not.

Re: Thoughts on a month with Devin

#72

Post by eitland »

DaveMcMartin wrote: post #54

This only reinforces my bias against AI agents. At this point, they are mostly just hype. I believe that for AI to replace a junior, we would need to achieve at least near-AGI, and we are far from that.

If by hype you mean that there isn't extreme real world value right here and right now, then I very much disagree.

Closing in on 20 years since I left school and for me AI is absolutely useful, right here and right now. It is really a bicycle for the mind:

It allows me to get much faster to where I want. (And like bicycles you will get a few crashes early on and possibly later as well, depending on how fast you move and how careful you are.)

I might be in some sweet spot where I am both old enough to know what is going on without using an AI but also young enough to pick up the use of AI relatively effortlessly.

If however by hype you mean that people still have overhyped expactations about the near future, then yes, I agree more and more.

Re: Thoughts on a month with Devin

#73

Post by frereubu »

CGamesPlay wrote: post #69

As someone who uses AI coding tools daily and has done a fair amount of experimentation with different approaches (though not Devin), I feel like this tracks pretty well. The problem is that Devin and other "agentic" approaches take on more than they can handle. The best AI coders are positioned as tools for developers, rather than replacements for them. Github Copilot is "a better tab complete". Sure, it's a neat de…

When I tried Copilot the "better tab complete" felt quite annoying, in that the constantly changing suggested completion kept dragging my focus away from what I was writing. That clearly doesn't happen for you. Was that something you got used to over time, or did that just not happen for you? There were elements of it I found useful, but I just couldn't get over the flickering of my attention from what I was doing to the suggested completions.

Edit: I also really want something that takes the existing codebase in the form of a VSCode project / GitHub repo and uses that as a basis for suggestions. Does Copilot do that now?

Re: Thoughts on a month with Devin

#74

Post by goosejuice »

frereubu wrote: post #73
CGamesPlay wrote: post #69

As someone who uses AI coding tools daily and has done a fair amount of experimentation with different approaches (though not Devin), I feel like this tracks pretty well. The problem is that Devin and other "agentic" approaches take on more than they can handle. The best AI coders are positioned as tools for developers, rather than replacements for them. Github Copilot is "a better tab complete". Sure, it's a neat de…

When I tried Copilot the "better tab complete" felt quite annoying, in that the constantly changing suggested completion kept dragging my focus away from what I was writing. That clearly doesn't happen for you. Was that something you got used to over time, or did that just not happen for you? There were elements of it I found useful, but I just couldn't get over the flickering of my attention from what I was doing to…

Copilot is also very slow. I'm surprised people use it to be honest. Just use Cursor.

Re: Thoughts on a month with Devin

#75

Post by [deleted] »

xmprt wrote: post #9

I think one of the big problems with Devin (and AI agents in general) is that they're only ever as good as they are. Sometimes their intelligence feels magical and they accomplish things within minutes that even mid level or senior software engineers would take a few hours to do. Other times, they make simple mistakes and no matter how much help you give, they run around in circles. A big quality that I value in juni…

[deleted]

Re: Thoughts on a month with Devin

#76

Post by rbren »

I'm one of the creators of OpenHands (fka OpenDevin). I agree with most of what's been said here, wrt to software agents in general.

We are not even close to the point where AI can "replace" a software engineer. Their code still needs to be reviewed and tested, at least as much as you'd scrutinize the code of a brand new engineer just out of boot camp. I've talked to companies who went all in on AI engineers, only to realize two months later that their codebase was rotting because no one was reviewing the changes.

But once you develop some intuition for how to use them, software agents can be a _massive_ boost to productivity. ~20% of the commits to the OpenHands codebase are now authored or co-authored by OpenHands itself. I especially love asking it to do simple, tedious things like fixing merge conflicts or failing linters. It's great at getting an existing PR over the line.

It's also important to keep in mind that these agents are literally improving on a _weekly_ basis. A few weeks ago we were at the top of the SWE-bench leaderboard; now there are half a dozen agents that have pulled ahead of us. And we're one launch away from leapfrogging back to the top. Exciting times!

https://github.com/All-Hands-AI/OpenHands

Re: Thoughts on a month with Devin

#77

Post by rfoo »

davedx wrote: post #6

One thing that surprised me a little is that there doesn't seem to be an "ask for help" escape hatch in it - it would work away for literally days on a task where any human would admit they were stuck? One of the more important features of agents is supposedly that they can stop and ask for human input when necessary. It seems it does do this for "hard stops" - like when it needed a human to setup API keys in their c…

Devin does ask for help when it can't do something. I think I have it asked me how to use a testing suite it had trouble running.

The problem is it really really hate asking for help if it had a skill issue, it would prefer running in circles than admitting it just can't do something.

Re: Thoughts on a month with Devin

#78

Post by Aeolun »

frereubu wrote: post #73
CGamesPlay wrote: post #69

As someone who uses AI coding tools daily and has done a fair amount of experimentation with different approaches (though not Devin), I feel like this tracks pretty well. The problem is that Devin and other "agentic" approaches take on more than they can handle. The best AI coders are positioned as tools for developers, rather than replacements for them. Github Copilot is "a better tab complete". Sure, it's a neat de…

When I tried Copilot the "better tab complete" felt quite annoying, in that the constantly changing suggested completion kept dragging my focus away from what I was writing. That clearly doesn't happen for you. Was that something you got used to over time, or did that just not happen for you? There were elements of it I found useful, but I just couldn't get over the flickering of my attention from what I was doing to…

Cursor tab does that. Or at least, it takes other open tabs into account when making suggestions.

Re: Thoughts on a month with Devin

#79

Post by Xmd5a »

noodletheworld wrote: post #60
Xmd5a wrote: post #51

Earlier quoted context omitted.

I had to look up MCST: it means Model-Centric Software Tools, as opposed to autonomous agents. Devin is closer to a long-running process that you can interact with as it is processing tasks, whereas Cursor is closer to a function call: once you've made the call, the only think you can do is wait for the result.

It stands for Monte Carlo search tree. Ie. Better outputs from models, not external tooling and prompt engineering. https://github.com/zz1358m/MCTS-AHD-master

Thanks for the correction, I guess I was lured by yet another LLM confabulation

Re: Thoughts on a month with Devin

#80

Post by Aeolun »

rfoo wrote: post #77
davedx wrote: post #6

One thing that surprised me a little is that there doesn't seem to be an "ask for help" escape hatch in it - it would work away for literally days on a task where any human would admit they were stuck? One of the more important features of agents is supposedly that they can stop and ask for human input when necessary. It seems it does do this for "hard stops" - like when it needed a human to setup API keys in their c…

Devin does ask for help when it can't do something. I think I have it asked me how to use a testing suite it had trouble running. The problem is it really really hate asking for help if it had a skill issue, it would prefer running in circles than admitting it just can't do something.

So they perfectly nailed the junior engineer. It’s just that that isn’t what people are looking for.
Post reply on HN