Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on a month with Devin

answer.ai

Post reply on HN
Print view View original

111–120 of 194 posts

Re: Thoughts on a month with Devin

#111

Post by pcwelder »

Most the problems you mentioned will likely be solved with the next iterations of Devin or similar product.

I can say that because I work daily with Claude as an agent over mcp, and the problems you mentioned feel very familiar.

Based on the type of the issues you mentioned, Devin isn't likely using o1 yet. A workflow like o1 for planning, Claude for Coding, o1 for review, etc., would work better.

The problems you mentioned: ssh-key issue unrelated to script, code not following existing patterns or themes, instructions not being followed, extra abstractions, etc., fall into that category.

Some of the issues are likely due to context length problem. For example, LLM doesn't work well with jupyter notebook because of extra junk in ipynb, which will likely remain a problem.

Re: Thoughts on a month with Devin

#112

Post by ErikBjare »

jebarker wrote: post #104
linsomniac wrote: post #100

Earlier quoted context omitted.

>burden seems much greater than... Because the burden is much lower than if you were authoring the same commit yourself without any automation?

Is that true? I'd like to think my commits are less burdensome to review than a fresh out of boot camp junior dev especially if all that's being done is fixing linter issues. Perhaps there's a small benefit, but doesn't seem like a major productivity boost.

A junior dev is not a good approximation of the strengths and weaknesses of these models.

Re: Thoughts on a month with Devin

#113

Post by verdverm »

davedx wrote: post #6

One thing that surprised me a little is that there doesn't seem to be an "ask for help" escape hatch in it - it would work away for literally days on a task where any human would admit they were stuck? One of the more important features of agents is supposedly that they can stop and ask for human input when necessary. It seems it does do this for "hard stops" - like when it needed a human to setup API keys in their c…

I think training it to do that would be the hard part.

- stopping is probably the easy part

- I assume this happens during RLFH phase

- Does the model simply stop or does it ask a question?

- You need a good response or interaction, depending on the query? So probably sets or decision trees of them, or agentic even? (chicken-egg problem?)

- This happens 10s of thousands of times, having humans do it, especially with coding, is probably not realistic

- Incumbents like M$ with Copilot may have an advantage in crafting a dataset

Re: Thoughts on a month with Devin

#114

Post by drewbug01 »

jebarker wrote: post #90
rbren wrote: post #76

I'm one of the creators of OpenHands (fka OpenDevin). I agree with most of what's been said here, wrt to software agents in general. We are not even close to the point where AI can "replace" a software engineer. Their code still needs to be reviewed and tested, at least as much as you'd scrutinize the code of a brand new engineer just out of boot camp. I've talked to companies who went all in on AI engineers, only to…

> code still needs to be reviewed and tested, at least as much as you'd scrutinize the code of a brand new engineer just out of boot camp > ..._massive_ boost to productivity. ~20% of the commits to the OpenHands codebase are now authored or co-authored by OpenHands itself. I'm having trouble reconciling these statements. Where does the productivity boost come from since that reviewing burden seems much greater than…

I suspect that many engineers do not expend significant energy on reviewing code; especially if the change is lengthy.

Re: Thoughts on a month with Devin

#115

Post by npilk »

This feels a bit like AI image generation in 2022. The fact that it works at all is pretty mindblowing, and sometimes it produces something really good, but most of the time there are obvious mistakes, errors, etc. Of course, it only took a couple more years to get photorealistic image outputs.

A lot of commenters here seem very quick to write off Devin / similar ideas permanently. But I'd guess in a few years the progress will be remarkable.

One stubborn problem – when I prompt Midjourney, what I get back is often very high-quality, but somehow different than what I expected. In other words, I wouldn't have been able to describe what I wanted, but once I see the output I know it's not quite right. I suspect tools like this will run into similar issues. Maybe there will be features that can help users 'iterate' quickly.

Re: Thoughts on a month with Devin

#116

Post by jebarker »

lars512 wrote: post #105
jebarker wrote: post #90

Earlier quoted context omitted.

> code still needs to be reviewed and tested, at least as much as you'd scrutinize the code of a brand new engineer just out of boot camp > ..._massive_ boost to productivity. ~20% of the commits to the OpenHands codebase are now authored or co-authored by OpenHands itself. I'm having trouble reconciling these statements. Where does the productivity boost come from since that reviewing burden seems much greater than…

There's often a lot of small fixes that not time efficient to do, but a solution is not much code and is quick to verify. If the cost is small to setting a coding agent (e.g. aider) on a task, seeing if it reaches a quick solution, and just aborting if it spins out, you can solve a subset of these types of issues very quickly, instead of leaving them in issue tracking to grow stale. That lets you up the polish on you…

Thanks - this all makes sense - I still don't feel like this would constitute a massive productivity boost in most cases, since it's not fixing time consuming major issues. But I can see how it's nice to have.

Re: Thoughts on a month with Devin

#117

Post by jamalaramala »

> Even more concerning was Devin’s tendency to press forward with tasks that weren’t actually possible. (...)

> Devin spent over a day attempting various approaches and hallucinating features that didn’t exist.

One of the big problems of GenAI is its inability to know what they don't know.

Because of that, they don't ask clarifying questions.

Humans, in the same situation, would spend a lot of time learning before they could be truly productive.

Re: Thoughts on a month with Devin

#118

Post by falcor84 »

thegeomaster wrote: post #59
DaveMcMartin wrote: post #54

This only reinforces my bias against AI agents. At this point, they are mostly just hype. I believe that for AI to replace a junior, we would need to achieve at least near-AGI, and we are far from that.

Exactly. LLMs are gullible . They will believe anything you tell them, including incorrect things they have told themselves. This amplifies errors greatly, because they don't have the capacity to step back and try a different approach, or introspect why they failed. They need actual guidance from somebody with much common sense; if let loose in the world, they mostly just spin around in circles because they don't hav…

A regular single-pass LLM indeed cannot step back, but newer ones like o1/o3/Marco-o1/QwQ can, and a larger agentic system composed of multiple LLMs definitely can. There is no "fundamental" limitation here. And once we start training these larger systems from the ground up via full reinforcement learning (rather than composing existing models), the sky's the limit. I'd be very bullish about Deepmind, once they fully enter this race.

Re: Thoughts on a month with Devin

#119

Post by neves »

Do you have good references about using AI coding assistants?

Techniques of prompt engineering help a lot, but I really think there will be created a body of knowledge about how to use, what's the good contexts of use, and good heuristics. They are a valuable tool, but I feel it's possible to extract more value.

Re: Thoughts on a month with Devin

#120

Post by mattnewton »

frereubu wrote: post #73
CGamesPlay wrote: post #69

As someone who uses AI coding tools daily and has done a fair amount of experimentation with different approaches (though not Devin), I feel like this tracks pretty well. The problem is that Devin and other "agentic" approaches take on more than they can handle. The best AI coders are positioned as tools for developers, rather than replacements for them. Github Copilot is "a better tab complete". Sure, it's a neat de…

When I tried Copilot the "better tab complete" felt quite annoying, in that the constantly changing suggested completion kept dragging my focus away from what I was writing. That clearly doesn't happen for you. Was that something you got used to over time, or did that just not happen for you? There were elements of it I found useful, but I just couldn't get over the flickering of my attention from what I was doing to…

I would try cursor. It’s pretty good at copy pasting the relevant parts of the codebase in and out of the chat window. I have the tab autocomplete disabled.
Post reply on HN