Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on a month with Devin

answer.ai

Post reply on HN
Print view View original

101–110 of 194 posts

Re: Thoughts on a month with Devin

#101

Post by kgilpin »

likium wrote: post #36
ianbutler wrote: post #23

Earlier quoted context omitted.

I've recently had several enterprise level conversations with different companies and what we're being asked for is specifically the simpler approach. I think that is the level of risk they're willing to tolerate and it will still ameliorate a real issue for them. The key here is my product is no worse positioned to do more things if and when the time comes, but building a solid foundation and trust, and not having t…

> You earn the right to take on bigger challenges and no one in this space has earned the right yet including us. Not entirely. We're in interesting times where products with better models can suddenly leapfrog and displace even current upstarts. Cursor won over Copilot from leveraging Claude Sonnet 3.5. They didn't "earn the right". Improvements with models will help those with the existing infrastructure that can b…

I just want to note that Copilot is multi model now and can also run Sonnet.

Re: Thoughts on a month with Devin

#103

Post by verdverm »

goosejuice wrote: post #93
pindab0ter wrote: post #88

Earlier quoted context omitted.

Cursor requires you to use their specific IDE though, doesn't it? With Copilot I don't have to switch contexts as it lives in my Jetbrains IDE.

It's just vscode. I greatly prefer vim but the difference between vim + ai tools and cursor is just a no brainer in terms of productivity. Cursor isn't without problems but it's leagues ahead of the competition in my opinion.

I've been tempted to try Cursor because of vocal fans like yourself. Then I went to their website and forums yesterday. I am no longer tempted.

Re: Thoughts on a month with Devin

#104

Post by jebarker »

linsomniac wrote: post #100
jebarker wrote: post #90

Earlier quoted context omitted.

> code still needs to be reviewed and tested, at least as much as you'd scrutinize the code of a brand new engineer just out of boot camp > ..._massive_ boost to productivity. ~20% of the commits to the OpenHands codebase are now authored or co-authored by OpenHands itself. I'm having trouble reconciling these statements. Where does the productivity boost come from since that reviewing burden seems much greater than…

>burden seems much greater than... Because the burden is much lower than if you were authoring the same commit yourself without any automation?

Is that true? I'd like to think my commits are less burdensome to review than a fresh out of boot camp junior dev especially if all that's being done is fixing linter issues. Perhaps there's a small benefit, but doesn't seem like a major productivity boost.

Re: Thoughts on a month with Devin

#105

Post by lars512 »

jebarker wrote: post #90
rbren wrote: post #76

I'm one of the creators of OpenHands (fka OpenDevin). I agree with most of what's been said here, wrt to software agents in general. We are not even close to the point where AI can "replace" a software engineer. Their code still needs to be reviewed and tested, at least as much as you'd scrutinize the code of a brand new engineer just out of boot camp. I've talked to companies who went all in on AI engineers, only to…

> code still needs to be reviewed and tested, at least as much as you'd scrutinize the code of a brand new engineer just out of boot camp > ..._massive_ boost to productivity. ~20% of the commits to the OpenHands codebase are now authored or co-authored by OpenHands itself. I'm having trouble reconciling these statements. Where does the productivity boost come from since that reviewing burden seems much greater than…

There's often a lot of small fixes that not time efficient to do, but a solution is not much code and is quick to verify.

If the cost is small to setting a coding agent (e.g. aider) on a task, seeing if it reaches a quick solution, and just aborting if it spins out, you can solve a subset of these types of issues very quickly, instead of leaving them in issue tracking to grow stale. That lets you up the polish on your work.

That's still quite a different story to having it do the core, most important part of your work. That feels a little further away. One of the challenges is the scout rule, the refactoring alongside change that makes the codebase nicer. I feel like today it's easier to get a correct change that slightly degrades codebase quality, than one that maintains it.

Re: Thoughts on a month with Devin

#106

Post by kgilpin »

kromem wrote: post #37
huijzer wrote: post #32

I’ve used Cursor a lot and the conclusion doesn’t surprise me. I feel like I’m the one *forcing* the system in a certain direction and sometimes an LLM gives a small snippet of useful code. Sometimes it goes in the wrong direction and I have to abort the suggestion and force it into another direction. For me, the main benefit is having a typing assistant which can save me from typing one line here and there. Especial…

Weird. I have such a different experience with Cursor. Most changes occur with a quick back and forth about top level choices in chat. Followed with me grabbing appropriate interfaces and files for context so Sonnet doesn't hallucinate API, and then code that I'll glance over and around half the time suggest one or more further changes. It's been successful enough I'm currently thinking of how to adjust best practice…

This is what’s needed to get the most out of these tools. You understand deeply how the tool works and so you’re able to optimize its inputs in order to get good results.

This puts you in the top echelon of developers using AI assisted coding. Most developers don’t have this deep of an understanding and so they don’t get results as good as yours.

So there’s a big question here for AI tool vendors. Is AI assisted coding a power tool for experts, or is it a tool for the “Everyman” developer that’s easy to use?

Usage data shows that the most adopted AI coding tool is still ChatGPT, followed by Copilot (even if you’d think it’s Cursor from reading HN :-))

Re: Thoughts on a month with Devin

#107

Post by mousetree »

davedx wrote: post #14
coffeebeqn wrote: post #8

Earlier quoted context omitted.

LLMs can create infinite worlds in the error message it’s receiving. It probably needs some outside signal to stop and re-assess. I don’t think LLMs have any ability to reason if they’re lost in their own world on their own. They’ll just keep creating new less and less coherent context for themselves

For sure - but if I'm paying for a tool like Devin then I'd expect the infrastructure around it to do things like stop it if it looks like that has happened. What you often see with agentic systems is that there's an agent whose role is to "orchestrate", and that's the kind of thing the orchestrator would do: every 10 minutes or so, check the output and elapsed time and decide if the "developer" agent needs a reality…

How would it decide if it needs a reality check? Would the thing checking have the same limitations?

Re: Thoughts on a month with Devin

#109

Post by verdverm »

fullstackchris wrote: post #96
dysoco wrote: post #94

Earlier quoted context omitted.

I'm confused here, aren't agents/assistants basically wrappers over LLMs or tools that interact with them as well? Devin seems to be in this category.

I recommend you look at tools like Aider or Codebuff... sure they need to call some LLM at some point (could be your own, could be external), but the key thing that they are doing complex modifications of source code using things like treesitter -> i.e. you don't rely directly on the LLM modifying code, but the LLM using trees to modify the code. See in Aider's sourcecode: https://github.com/Aider-AI/aider/tree/main/…

Can Aider sit inside VS Code, understand what files I have open, and use them as context? Their docs lead me to say no, that they are an inline chat/completion experience

Re: Thoughts on a month with Devin

#110

Post by macNchz »

viraptor wrote: post #56
halfmatthalfcat wrote: post #31

Earlier quoted context omitted.

I disagree. If anything, CS degrees have proven time and time again they aren't translatable into software development (which is why there's an entire degree field called Software Engineering emerging). If anything, my gut says that the CS concepts are very easy for LLMs to recall and will be the first things replaced (if ever) by AI. Software engineer{ing,s} (project construction, integrations, scaling, organization…

> the CS concepts are very easy for LLMs to recall They're easy to recall, but you have to know what to recall in the first place. Or even know enough of the territory to realise there's something to recall. Without enough background, you'll get a whole set of amazing tools that you have no idea what to do with. For example, you may be able to write a long description of your problem with some ideas how to steer the…

To this point, I use AI programming assistants pretty heavily and find very frequently that they will write extremely inefficient or oddly baroque implementations of what I’m asking for in their first pass, that appear as if they don’t have the “knowledge” or ability to do it better, but then they can be prodded to re-do it very easily. Frequently I look at some generated code and write back the most cursory feedback like “looks o(n^2) can you make more efficient” or “use pointers instead of nested loops” or “how about using X approach” and it will often produce something dramatically better than the initial effort. For now at least I think these tools are still most powerful in the hands of experts. (I am a self-taught programmer but have a fair bit of experience)
Post reply on HN