Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on a month with Devin

answer.ai

Post reply on HN
Print view View original

161–170 of 194 posts

Re: Thoughts on a month with Devin

#161

Post by lolinder »

senordevnyc wrote: post #157
lolinder wrote: post #134

Earlier quoted context omitted.

I remember when I was first getting started in the industry the big fear of the time was that offshoring was going to take all of our jobs and drive down the salaries of those that remained. In fact the opposite happened: it was in the next 10 years that salaries ballooned and tech had a hiring bubble. Companies always want to reduce staff and bad companies always try to do so before the solution has really proven it…

But having deep experience with these tools over many years, I'm very confident... No one has had deep experience with these tools for any amount of time, let alone many years. They're literally just now hitting the market and are rapidly expanding their capabilities. We're at a fundamentally different place than we were just twelve months ago, and there's no reason to think 2025 will be any different.

I was building things with GPT-2 in 2019. I have as much experience engineering with them as anyone who wasn't an AI researcher before then.

And no, we're not at a fundamentally different place than we were just 12 months ago. The last 12 months had much slower growth than the 12 months before that, which had slower growth than the 12 months before that. And in the end these tools have the same weaknesses that I saw in GPT-2, just to a lesser degree.

The only aspect in which we are in a fundamentally different place is that the hype has gone through the roof. The tools themselves are better, but not fundamentally different.

Re: Thoughts on a month with Devin

#162

Post by verdverm »

goosejuice wrote: post #155
verdverm wrote: post #103

Earlier quoted context omitted.

I've been tempted to try Cursor because of vocal fans like yourself. Then I went to their website and forums yesterday. I am no longer tempted.

Because of the complaints? If so, yeah I get it. I'm there amongst them. It's kind of like Tesla FSD. There are often setbacks in releases and they definitely need to work on their communication with the community. That said, for the current price it's still worth any misgivings.

The price is one of the issues I have with this space more generally.

I do not want to pay $20/m for a capped experience

I want to pay $10/m to support development, and pay for my AI usage on my own, per request, by choosing my own model and provider

If I was going to shell out money, it would be for Copilot, not Cursor. I prefer my AI to be a side dish, not the main course or core experience

Re: Thoughts on a month with Devin

#163

Post by verdverm »

rob137 wrote: post #142
verdverm wrote: post #103

Earlier quoted context omitted.

I've been tempted to try Cursor because of vocal fans like yourself. Then I went to their website and forums yesterday. I am no longer tempted.

Can you say more?

pricing model, downtime, model support, pricing model, trying to take over the experience rather than assist within my experience. This last one is big, because Cursor wants to "reimagine" how developers work. The problem is the AIs are so far from being competent, they need to be kept on the sidelines and sub'd in occasionally, not be the quarterback. Oh, did I mention pricing model?

Re: Thoughts on a month with Devin

#164

Post by sherburt3 »

moffkalast wrote: post #65
epolanski wrote: post #57

Earlier quoted context omitted.

I'll add few things at which Cursor with Claude is better than us (at least in time/effort): - explaining code. Enter some legacy part of your code nobody understands, LLMs aren't limited to keeping few things in memory like us. Even if the code is very obfuscated and poorly written it can understand what it does and the purpose and suggest refactors to make it understandable - explaining and fixing bugs. Just the ot…

That first one has always felt super crazy to me, I've figured out what lots of "arcane magic, don't touch" type of functions genuinely do since LLMs have become a thing. Even if it's slightly wrong it's usually at least in the right ballpark so it gives you a very good starting point to work from. Almost everything is explainable now.

Agreed, AI has been a godsend for trying to understand snippets of perl code in our codebase that were basically unreadable before unless you were an expert.

Re: Thoughts on a month with Devin

#165

Post by gtsop »

meiraleal wrote: post #28
gtsop wrote: post #13

Honestly, i have been bitten so many times by LLM hallucinations when I work in parallel with the LLM, I wouldn't trust it autonomously running anything at all. If you have tried to use imaginary APIs, imaginary configuration and imaginary cli arguments, you know what I mean

I have been feeling LLM burnout and favoring code it all my self after a year of LLM assistance. When it gets things wrong it is too annoying. Like, I would get mad and start to curse it, shouting loud and in the chat.

Exactly this. At first started verbally abusing it untill it conformed, but i quickly realised that after the context gets very long it simply discards former instructions and abusing. So i get frustrated, toxic AND don't get my job done

Re: Thoughts on a month with Devin

#166

Post by gtsop »

energy123 wrote: post #16
gtsop wrote: post #13

Honestly, i have been bitten so many times by LLM hallucinations when I work in parallel with the LLM, I wouldn't trust it autonomously running anything at all. If you have tried to use imaginary APIs, imaginary configuration and imaginary cli arguments, you know what I mean

> If you have tried to use imaginary APIs, imaginary configuration and imaginary cli arguments, you know what I mean I see this comment a lot but I can't help but feel it's 4 weeks out of date. The version of o1 released on 2024-12-17 so rarely hallucinates when asked code questions of basic to medium difficulty and provided with good context and a well written prompt, in my experience. If the context window is sub-1…

o1 is way to slow to keep up with my flow of thinking in order to be of any help in the scenario i am describing

Re: Thoughts on a month with Devin

#168

Post by epicureanideal »

the_af wrote: post #149
s1mplicissimus wrote: post #135

Earlier quoted context omitted.

> Ok then why does nearly every company make people write code for interviews or do take home programming projects? For the same reason they put leetcode problems to "test" an applicants skill. Or have them write mergesort on a chalkboard by hand. It gives them a warm fuzzy feeling in the tummy because now they can say "we did something to check they are competent". Why, you ask? Well it's mostly impossible to come u…

Arguing about programming is not the point, in my opinion. When AI becomes able to do most non-programming tasks too, say design or solving open-ended problems (yeah, except in trivial cases it cannot -- for now) we can have this conversation again... I think saying "well, programming is not important, what matters is $THING" is a coping mechanism. Eventually AI will do $THING acceptably enough for the bean counters…

Also, because the economy is no longer based on competition, but is controlled by a bunch of industry specific oligopolies, even if the bean counters are wrong it won’t matter, because every other company will be similarly inefficient. Everybody loses, but the people in charge are too dumb to know. Our free market is currently broken.

Re: Thoughts on a month with Devin

#169

Post by thegeomaster »

falcor84 wrote: post #118
thegeomaster wrote: post #59

Earlier quoted context omitted.

Exactly. LLMs are gullible . They will believe anything you tell them, including incorrect things they have told themselves. This amplifies errors greatly, because they don't have the capacity to step back and try a different approach, or introspect why they failed. They need actual guidance from somebody with much common sense; if let loose in the world, they mostly just spin around in circles because they don't hav…

A regular single-pass LLM indeed cannot step back, but newer ones like o1/o3/Marco-o1/QwQ can, and a larger agentic system composed of multiple LLMs definitely can. There is no "fundamental" limitation here. And once we start training these larger systems from the ground up via full reinforcement learning (rather than composing existing models), the sky's the limit. I'd be very bullish about Deepmind, once they fully…

> And once we start training these larger systems from the ground up via full reinforcement learning (rather than composing existing models),

Agree with this totally.

I wouldn't call what the CoT models are doing exactly being able to step back - their "stepping back" still dumps tokens into the output, so it is still burdened with seeing all of these failed attempts as it searches for the right one. But my intuition on this can be wrong, and it's a much more advanced reasoning process than what "last-gen" (non-CoT) does, so I can see your point.

For an agentic system composed of multiple LLMs, I would strongly disagree if the LLMs are last-gen. In my experience, it is very hard to prompt a non-CoT LLM into rejecting an upstream assumption without making it paranoid and rejecting valid assumptions as well. This makes it hard to effectively create a robust agentic system that can self-correct.

I think that's different if the agents are o1-level, but I think it's hard to appreciate just how costly and slow doing this would be. Agents consume tokens like candy with all the back-and-forth, so a surprising number of tasks become economically infeasible.

(It seems everyone is waiting for an inference perf breakthrough that may or may not come.)

Re: Thoughts on a month with Devin

#170

Post by svieira »

rbren wrote: post #128
jebarker wrote: post #116

Earlier quoted context omitted.

Thanks - this all makes sense - I still don't feel like this would constitute a massive productivity boost in most cases, since it's not fixing time consuming major issues. But I can see how it's nice to have.

The bigger win comes not from saving keystrokes, but from saving you from a context switch. Merge conflicts are probably the biggest one for me. I put up a PR and move onto a new task. Someone approves, but now there are conflicts. I could switch off my task, spend 5-10 min remembering the intent of this PR and fixing the issues. Or I could just say "@openhands fix the merge conflicts" and move back to my new task.

The issue is that you still need to review the fixed PR (or someone else does) which means you just deferred the context switch, you didn't eliminate it. And if the fix is in a new commit, that's possible (whereas if it rebases you have to remember your old SHA).

Playing the other side, pipelining is real.

Post reply on HN