Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on a month with Devin

answer.ai

Post reply on HN
Print view View original

171–180 of 194 posts

Re: Thoughts on a month with Devin

#171

Post by svieira »

mousetree wrote: post #107
davedx wrote: post #14

Earlier quoted context omitted.

For sure - but if I'm paying for a tool like Devin then I'd expect the infrastructure around it to do things like stop it if it looks like that has happened. What you often see with agentic systems is that there's an agent whose role is to "orchestrate", and that's the kind of thing the orchestrator would do: every 10 minutes or so, check the output and elapsed time and decide if the "developer" agent needs a reality…

How would it decide if it needs a reality check? Would the thing checking have the same limitations?

Decision trees and random forests (funnily enough, this is not sarcasm).

Re: Thoughts on a month with Devin

#172

Post by ZephyrBlu »

the_af wrote: post #149
s1mplicissimus wrote: post #135

Earlier quoted context omitted.

> Ok then why does nearly every company make people write code for interviews or do take home programming projects? For the same reason they put leetcode problems to "test" an applicants skill. Or have them write mergesort on a chalkboard by hand. It gives them a warm fuzzy feeling in the tummy because now they can say "we did something to check they are competent". Why, you ask? Well it's mostly impossible to come u…

Arguing about programming is not the point, in my opinion. When AI becomes able to do most non-programming tasks too, say design or solving open-ended problems (yeah, except in trivial cases it cannot -- for now) we can have this conversation again... I think saying "well, programming is not important, what matters is $THING" is a coping mechanism. Eventually AI will do $THING acceptably enough for the bean counters…

When AI can do the software engineering tasks that require expertise outside of coding like system design, scoping problems, cross-team/domain work, etc then it will be AGI, at which point the fact that SWE jobs are automated would be the least of everyones worries.

The main problem I perceive with AI being able to do that kind of work is that it requires an unprecedented level of agency and context-gathering. Right now agents are very much like juniors in that they work in an insular, not collaborative, way.

Another big problem is that these higher level problems often require piecing together a lot of fragmented context. If the AI already had access to the information, sure, it would probably be able to achieve the task. But the hard bit is finding the information. Some logs here, some code there, a conversation with someone on a different team, etc. It's often a highly intuitive and tacit process, not easily explicitly defined. There's a reason that defining what a "Senior" is tends to be very difficult.

Re: Thoughts on a month with Devin

#173

Post by the_af »

ZephyrBlu wrote: post #172
the_af wrote: post #149

Earlier quoted context omitted.

Arguing about programming is not the point, in my opinion. When AI becomes able to do most non-programming tasks too, say design or solving open-ended problems (yeah, except in trivial cases it cannot -- for now) we can have this conversation again... I think saying "well, programming is not important, what matters is $THING" is a coping mechanism. Eventually AI will do $THING acceptably enough for the bean counters…

When AI can do the software engineering tasks that require expertise outside of coding like system design, scoping problems, cross-team/domain work, etc then it will be AGI, at which point the fact that SWE jobs are automated would be the least of everyones worries. The main problem I perceive with AI being able to do that kind of work is that it requires an unprecedented level of agency and context-gathering. Right…

> When AI can do the software engineering tasks that require expertise outside of coding like system design, scoping problems, cross-team/domain work, etc then it will be AGI

I think you're talking about the really general case, but in my opinion that's not as important. All that matters is that AI solutions manage (in the near future) to cover the average case -- where most engineers actually work -- in a mediocre but cost effective manner, for this to have huge repercussions on the job market and salaries.

> But the hard bit is finding the information. Some logs here, some code there, a conversation with someone on a different team, etc.

I've no problem believing they will become more and more successful at this. This is information retrieval which can be done faster by machines, and making sense of it all together is where advances in AI will need to happen. I think there's a high chance they'll happen eventually, at least in a way that's enough to cobble together projects that will make the leadership happy (maybe after some review/adjustment by a few human experts they retain?). They do not even have to be particularly successful -- how many human-populated engineering projects succeed, anyway?

Re: Thoughts on a month with Devin

#174

Post by energy123 »

gtsop wrote: post #166
energy123 wrote: post #16

Earlier quoted context omitted.

> If you have tried to use imaginary APIs, imaginary configuration and imaginary cli arguments, you know what I mean I see this comment a lot but I can't help but feel it's 4 weeks out of date. The version of o1 released on 2024-12-17 so rarely hallucinates when asked code questions of basic to medium difficulty and provided with good context and a well written prompt, in my experience. If the context window is sub-1…

o1 is way to slow to keep up with my flow of thinking in order to be of any help in the scenario i am describing

How are you using LLMs? With o1 I've switched to spelling out in lots of details what I want, then asking it it to one shot the full file, so with this approach the wait time has been acceptable.

Re: Thoughts on a month with Devin

#175

Post by jondwillis »

freddref wrote: post #49
noodletheworld wrote: post #17

Those “how I feel about Devin after using it” comments at the bottom are damning, when you compare them to the user testimonials of people using cursor. Seems to me that agents just aren’t the answer people want them to be, just a hype wave obscuring real progress in other areas (eg. MCST) because they’re easy to implement. …but really, if things are easy to implement, at this point, you have to ask why they haven’t…

How is Devin different from cursor? I recently used cursor and it has felt very capable in implementing tasks across files. I get that cursor is an IDE but it's ai functionality feels very agentic.. where do you draw the line?

Cursor Composer (both "normal" and "agent" mode) fit the colloquial definition of agent, for sure.

Re: Thoughts on a month with Devin

#176

Post by monophonica »

verdverm wrote: post #103
goosejuice wrote: post #93

Earlier quoted context omitted.

It's just vscode. I greatly prefer vim but the difference between vim + ai tools and cursor is just a no brainer in terms of productivity. Cursor isn't without problems but it's leagues ahead of the competition in my opinion.

I've been tempted to try Cursor because of vocal fans like yourself. Then I went to their website and forums yesterday. I am no longer tempted.

It is worth trying.

It is just a fashion choice though with UI.

Personally, I just prefer the chat interface directly with no Cursor UI.

For me, the best way is to write my prompt in a txt file, away from anything to do with LLMs. The bottleneck is not the update of the files like Cursor is good at.

The bottleneck is the clarity of my thoughts.

I looked at your website.

How to get past Barry Schwartz ideas is the main problem that we face in 2025.

The Godel, Escher, Bach stuff to me is just nonsense. As a huge Bach fan boy it is from when Bach was massively overrated in cultural importance.

Hierarchy Theory? How about O-information?

Doesn't seem the O-information wiki entry exists, yet.

Re: Thoughts on a month with Devin

#178

Post by senordevnyc »

lolinder wrote: post #161
senordevnyc wrote: post #157

Earlier quoted context omitted.

But having deep experience with these tools over many years, I'm very confident... No one has had deep experience with these tools for any amount of time, let alone many years. They're literally just now hitting the market and are rapidly expanding their capabilities. We're at a fundamentally different place than we were just twelve months ago, and there's no reason to think 2025 will be any different.

I was building things with GPT-2 in 2019. I have as much experience engineering with them as anyone who wasn't an AI researcher before then. And no, we're not at a fundamentally different place than we were just 12 months ago. The last 12 months had much slower growth than the 12 months before that, which had slower growth than the 12 months before that. And in the end these tools have the same weaknesses that I saw…

It’s genuinely difficult to take seriously a claim that coding using Sonnet has “the same weaknesses” as GPT-2, which was effectively useless for the task. It’s like suggesting that a flamethrower has the same weaknesses as a matchstick because they both can be put out by water.

We’ll have to agree to disagree about whether the last 12 months has had as much innovation as the preceding 12 months. We started 2024 with no models better than GPT-4, and we ended the year with multiple open source models that beat GPT-4 and can run on your laptop, not to mention a bunch of models that trounce it. Plus tons of other innovations, dramatically cheaper training and inference costs, reasoning models, expanded multi-modal capabilities, etc, etc.

I’m guessing you’ve already seen and dismissed it, but in case you’re interested in an overview, this is a good one: https://simonwillison.net/2024/Dec/31/llms-in-2024/

Re: Thoughts on a month with Devin

#179

Post by dimitri-vs »

senordevnyc wrote: post #178
lolinder wrote: post #161

Earlier quoted context omitted.

I was building things with GPT-2 in 2019. I have as much experience engineering with them as anyone who wasn't an AI researcher before then. And no, we're not at a fundamentally different place than we were just 12 months ago. The last 12 months had much slower growth than the 12 months before that, which had slower growth than the 12 months before that. And in the end these tools have the same weaknesses that I saw…

It’s genuinely difficult to take seriously a claim that coding using Sonnet has “the same weaknesses” as GPT-2, which was effectively useless for the task. It’s like suggesting that a flamethrower has the same weaknesses as a matchstick because they both can be put out by water. We’ll have to agree to disagree about whether the last 12 months has had as much innovation as the preceding 12 months. We started 2024 with…

I'm paying for o1-pro (just for one month) and have been using LLMs since GPT-2 (via AI Dungeon). Progress is absolutely flattering when you're looking at practical applications versus benchmarks.

o1 is actually surprisingly "meh" and I just don't see how they can justify the price when sonnet 3.5 latest is almost as good, 10x as fast and doesn't even have "reasoning".

I'm spending half my day every day for the past few years using LLMs in one way or another. They still confidently (and unpredictability) hallucinate, even o1. They have no memory, can't build up experience, performance rapidly degrades with long conversations, etc.

I'm not saying progress isn't being made, but the rate of progress is definitely slowing.

Re: Thoughts on a month with Devin

#180

Post by dimitri-vs »

nyarlathotep_ wrote: post #130
the_af wrote: post #124

Earlier quoted context omitted.

> The best AI coders are positioned as tools for developers, rather than replacements for them. I agree with this. However, we must not delude ourselves and understand that corporate is pushing for replacement . So there will be a big push to improve on tools like Devin. This is not a conspiracy theory, in many companies (my wife's, for example) they are openly stating this: we are going to reduce (aka "lay off") the…

> I wonder how many of us, here, understand that many jobs are going away if/when this works out for the companies. And the usual coping mechanism, "it will only be for low hanging fruit", "it will never happen to me because my $SKILL is not replaceable", will eventually not save you. Sure, if you are a unique expert on a unique field, but many of us don't have that luxury. And it'll be used to drive down salaries, t…

Is spending 4 years of your life on education that will likely only be 10-20% applicable to your job any less bizarre? It's just another hoop employers want to see you capable of jumping.

If you ignore the syntax programming is just writing detailed instructions. Just because AI is able to translate English to code doesn't mean the 100s of decisions that need to be made go away. Someone still needs to write very detailed instructions even if they are in English and it sure isn't going to be the people sitting in meetings all day.

And let's pretend that I can now be 10x more productive with AI. Great, now I can ship 10x more features in the same timeframe and nothing changes - the development backlog is literally infinite. There are always more features or bugs to work on.

Post reply on HN