Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on a month with Devin

answer.ai

Post reply on HN
Print view View original

61–70 of 194 posts

Re: Thoughts on a month with Devin

#61

Post by llamaimperative »

kkaatii wrote: post #43
xmprt wrote: post #9

I think one of the big problems with Devin (and AI agents in general) is that they're only ever as good as they are. Sometimes their intelligence feels magical and they accomplish things within minutes that even mid level or senior software engineers would take a few hours to do. Other times, they make simple mistakes and no matter how much help you give, they run around in circles. A big quality that I value in juni…

And when working with people it's fairly easy to intervene and improve when needed. I think the current working model with LLMs is definitely suboptimal when we cannot confine their solution space AND where they should apply a solution precisely, and timely.

It’s also often possible to know what a human will be bad at before they start. This allows you to delegate tasks better or vary the level of pre-work you do before getting started. This is pretty unpredictable with LLMs still.

Re: Thoughts on a month with Devin

#62

Post by llamaimperative »

Melomomololo wrote: post #38
noodletheworld wrote: post #17

Those “how I feel about Devin after using it” comments at the bottom are damning, when you compare them to the user testimonials of people using cursor. Seems to me that agents just aren’t the answer people want them to be, just a hype wave obscuring real progress in other areas (eg. MCST) because they’re easy to implement. …but really, if things are easy to implement, at this point, you have to ask why they haven’t…

Agents are really new and would solve plenty of annoying things. When I code with Claude, I have to copy paste files around. But everything we do in AI is new and outdated a few weeks ago. Claude is really good but blocks you in 1-3h for a bit due to context length. That type of issues will be solved. And local coding models are super fast on a 4090 already. Imagine a small project digits on your desktop were you all…

Have you used Cursor, which GP actually refers to?

Re: Thoughts on a month with Devin

#63

Post by nejsjsjsbsb »

meiraleal wrote: post #28
gtsop wrote: post #13

Honestly, i have been bitten so many times by LLM hallucinations when I work in parallel with the LLM, I wouldn't trust it autonomously running anything at all. If you have tried to use imaginary APIs, imaginary configuration and imaginary cli arguments, you know what I mean

I have been feeling LLM burnout and favoring code it all my self after a year of LLM assistance. When it gets things wrong it is too annoying. Like, I would get mad and start to curse it, shouting loud and in the chat.

I mainly use it as a typing assist. If it suggests ahead what I was thinking it saves time.

Re: Thoughts on a month with Devin

#64

Post by paradite »

timabdulla wrote: post #53
paradite wrote: post #35

I also wrote my first impressions on Devin, more focused on the user experience and analysis of its capabilities (with lots of screenshots): https://thegroundtruth.substack.com/p/devin-first-impression...

Your take seems much more positive than theirs. What do you think the key differences are between your experience and the one here?

One possible reason is that I'm using popular tech stacks (Next.js, HTML/JS for demo website and SDK). No niche frameworks or tools like nbdev (I've never heard of that).

Also I've been prompting ChatGPT and Claude for over a year, that might help with communicating with Devin.

Re: Thoughts on a month with Devin

#65

Post by moffkalast »

epolanski wrote: post #57
huijzer wrote: post #32

I’ve used Cursor a lot and the conclusion doesn’t surprise me. I feel like I’m the one *forcing* the system in a certain direction and sometimes an LLM gives a small snippet of useful code. Sometimes it goes in the wrong direction and I have to abort the suggestion and force it into another direction. For me, the main benefit is having a typing assistant which can save me from typing one line here and there. Especial…

I'll add few things at which Cursor with Claude is better than us (at least in time/effort): - explaining code. Enter some legacy part of your code nobody understands, LLMs aren't limited to keeping few things in memory like us. Even if the code is very obfuscated and poorly written it can understand what it does and the purpose and suggest refactors to make it understandable - explaining and fixing bugs. Just the ot…

That first one has always felt super crazy to me, I've figured out what lots of "arcane magic, don't touch" type of functions genuinely do since LLMs have become a thing.

Even if it's slightly wrong it's usually at least in the right ballpark so it gives you a very good starting point to work from. Almost everything is explainable now.

Re: Thoughts on a month with Devin

#66

Post by motbus3 »

I saw few people around testing and it is quite disappointing. Sometimes a task might take forever and deliver a bad result or fail completely.

It seems it is targeting few specific problems and whatever else is just too hard. I also think that, thought it is expensive, it is cheap for the technology behind it and it won't be able to keep that price for long

Re: Thoughts on a month with Devin

#68

Post by bsenftner »

At some point people are going to realize that using these LLM AIs is a communications problem, and by that I mean the reason various attempts to use them fail is because they are not being effectively told what to do, vague and implied requests are not enough for a inhuman statistical construct to grasp what you're asking without clearer more details and more specific instructions.

Re: Thoughts on a month with Devin

#69

Post by CGamesPlay »

As someone who uses AI coding tools daily and has done a fair amount of experimentation with different approaches (though not Devin), I feel like this tracks pretty well. The problem is that Devin and other "agentic" approaches take on more than they can handle. The best AI coders are positioned as tools for developers, rather than replacements for them.

Github Copilot is "a better tab complete". Sure, it's a neat demo that it can produce a fast inverse square root, but the real utility is that it completes repetitive code. It's like having a dynamic snippet library always available that I never have to configure.

Aider is the next step up the abstraction ladder. It can edit in more locations than just the current cursor position, so it can perform some more high-level edit operations. And although it also uses a smarter model than Copilot, it still isn't very "smart" at the end of the day, and will hallucinate functions and make pointless changes when you give it a problem to solve.

Re: Thoughts on a month with Devin

#70

Post by epolanski »

moffkalast wrote: post #65
epolanski wrote: post #57

Earlier quoted context omitted.

I'll add few things at which Cursor with Claude is better than us (at least in time/effort): - explaining code. Enter some legacy part of your code nobody understands, LLMs aren't limited to keeping few things in memory like us. Even if the code is very obfuscated and poorly written it can understand what it does and the purpose and suggest refactors to make it understandable - explaining and fixing bugs. Just the ot…

That first one has always felt super crazy to me, I've figured out what lots of "arcane magic, don't touch" type of functions genuinely do since LLMs have become a thing. Even if it's slightly wrong it's usually at least in the right ballpark so it gives you a very good starting point to work from. Almost everything is explainable now.

I can relate, I have been genuinely amazed more than once by how it could "understand" some very complex code nobody dared to touch like you mention.
Post reply on HN