Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Thoughts on a month with Devin

answer.ai

Post reply on HN
Print view View original

151–160 of 194 posts

Re: Thoughts on a month with Devin

#151

Post by lolinder »

the_af wrote: post #148
lolinder wrote: post #134

Earlier quoted context omitted.

I remember when I was first getting started in the industry the big fear of the time was that offshoring was going to take all of our jobs and drive down the salaries of those that remained. In fact the opposite happened: it was in the next 10 years that salaries ballooned and tech had a hiring bubble. Companies always want to reduce staff and bad companies always try to do so before the solution has really proven it…

I think it's qualitatively different this time. Unlike with offshoring, this is a technological solution, which understandably is received more enthusiastically on HN. I get it. It's interesting as tech! And it's achieved remarkable things. But unlike with offshoring (which is a people thing) or magical NOCODE/CASE/etc "solutions", it seems the consensus is that AI coding assistants will eventually get there. At leas…

> But unlike with offshoring (which is a people thing) or magical NOCODE/CASE/etc "solutions", it seems the consensus is that AI coding assistants will eventually get there.

There's no consensus to that point. There are a few loud hype artists, most of whom are employed in AI and have so have conflicts of interest and also are pre-filtered to the true believers. Their logic is basically "See this trend? Trends continue, so this is inevitable!"

That's bad logic. Trends do not always continue, they often slow or reverse, and this one is showing all signs of doing so already. OpenAI has come straight out and said that they don't expect to see another jump like GPT-3 to 4, and have resorted to throwing more tokens at the problems, which works with diminishing returns. I do not expect to see a return to the rapid growth we had for a year or two there.

> PS: I am offhsoring (in a way). I'm not based in the US but I work remotely for a US company.

Yes, and this is a good example: there's a place for offshoring, but it didn't replace US devs. The same thing will happen here.

Re: Thoughts on a month with Devin

#152

Post by bufferoverflow »

rbren wrote: post #76

I'm one of the creators of OpenHands (fka OpenDevin). I agree with most of what's been said here, wrt to software agents in general. We are not even close to the point where AI can "replace" a software engineer. Their code still needs to be reviewed and tested, at least as much as you'd scrutinize the code of a brand new engineer just out of boot camp. I've talked to companies who went all in on AI engineers, only to…

We've seen exponential improvements in LLM's coding abilities. Went from almost useless to somewhat useful in like two years.

Claude 3.5 is not bad really. I wanted to do a side project that has been on my mind for a few years, and Claude coded it in like 30 seconds.

So to say "we are not even close" seems strange. If LLMs continue to improve, they will be comparable to mid level developers in 2-3 years, senior developers in 4-5 years.

Re: Thoughts on a month with Devin

#153

Post by Zanfa »

bufferoverflow wrote: post #152
rbren wrote: post #76

I'm one of the creators of OpenHands (fka OpenDevin). I agree with most of what's been said here, wrt to software agents in general. We are not even close to the point where AI can "replace" a software engineer. Their code still needs to be reviewed and tested, at least as much as you'd scrutinize the code of a brand new engineer just out of boot camp. I've talked to companies who went all in on AI engineers, only to…

We've seen exponential improvements in LLM's coding abilities. Went from almost useless to somewhat useful in like two years. Claude 3.5 is not bad really. I wanted to do a side project that has been on my mind for a few years, and Claude coded it in like 30 seconds. So to say "we are not even close" seems strange. If LLMs continue to improve, they will be comparable to mid level developers in 2-3 years, senior devel…

> So to say "we are not even close" seems strange. If LLMs continue to improve, they will be comparable to mid level developers in 2-3 years, senior developers in 4-5 years.

These sorts of things can’t be extrapolated. It could be 6-months, it could be a local maxima / dead end that’ll take another breakthrough in 10 years like transformers were. See self-driving cars.

Re: Thoughts on a month with Devin

#155

Post by goosejuice »

verdverm wrote: post #103
goosejuice wrote: post #93

Earlier quoted context omitted.

It's just vscode. I greatly prefer vim but the difference between vim + ai tools and cursor is just a no brainer in terms of productivity. Cursor isn't without problems but it's leagues ahead of the competition in my opinion.

I've been tempted to try Cursor because of vocal fans like yourself. Then I went to their website and forums yesterday. I am no longer tempted.

Because of the complaints? If so, yeah I get it. I'm there amongst them. It's kind of like Tesla FSD. There are often setbacks in releases and they definitely need to work on their communication with the community. That said, for the current price it's still worth any misgivings.

Re: Thoughts on a month with Devin

#156

Post by Yenrabbit »

exo-cortex wrote: post #21

I remain sceptical about the "Planet Tracker"-task. The task was to debunk claims about historical positions of Jupiter and Saturn. If the task was to find those planets were NOT in a certain (claimed) position an erroneous program would still appear to "debunk" the claims. Did they check if Devin's code's calculated positions were actually correct? Did they check in some NASA-database? If Devin gave arbitrary positi…

I was able to read the code it wrote, and check that (as hoped) it was using a good existing library to do the heavy lifting. And I had it make plots that I could visually use to check that the values were 'reasonable'. The value in that case was simply that I didn't have to leave the couch and write the code myself (although if the result was actually needed for anything more important than a smug 'i thought so' confirmation I would still have taken over and validated it kore carefully).

Re: Thoughts on a month with Devin

#157

Post by senordevnyc »

lolinder wrote: post #134
the_af wrote: post #124

Earlier quoted context omitted.

> The best AI coders are positioned as tools for developers, rather than replacements for them. I agree with this. However, we must not delude ourselves and understand that corporate is pushing for replacement . So there will be a big push to improve on tools like Devin. This is not a conspiracy theory, in many companies (my wife's, for example) they are openly stating this: we are going to reduce (aka "lay off") the…

I remember when I was first getting started in the industry the big fear of the time was that offshoring was going to take all of our jobs and drive down the salaries of those that remained. In fact the opposite happened: it was in the next 10 years that salaries ballooned and tech had a hiring bubble. Companies always want to reduce staff and bad companies always try to do so before the solution has really proven it…

But having deep experience with these tools over many years, I'm very confident...

No one has had deep experience with these tools for any amount of time, let alone many years. They're literally just now hitting the market and are rapidly expanding their capabilities. We're at a fundamentally different place than we were just twelve months ago, and there's no reason to think 2025 will be any different.

Re: Thoughts on a month with Devin

#158

Post by senordevnyc »

lolinder wrote: post #151
the_af wrote: post #148

Earlier quoted context omitted.

I think it's qualitatively different this time. Unlike with offshoring, this is a technological solution, which understandably is received more enthusiastically on HN. I get it. It's interesting as tech! And it's achieved remarkable things. But unlike with offshoring (which is a people thing) or magical NOCODE/CASE/etc "solutions", it seems the consensus is that AI coding assistants will eventually get there. At leas…

> But unlike with offshoring (which is a people thing) or magical NOCODE/CASE/etc "solutions", it seems the consensus is that AI coding assistants will eventually get there. There's no consensus to that point. There are a few loud hype artists, most of whom are employed in AI and have so have conflicts of interest and also are pre-filtered to the true believers. Their logic is basically "See this trend? Trends contin…

Trends do not always continue, they often slow or reverse, and this one is showing all signs of doing so already. OpenAI has come straight out and said that they don't expect to see another jump like GPT-3 to 4, and have resorted to throwing more tokens at the problems, which works with diminishing returns. I do not expect to see a return to the rapid growth we had for a year or two there.

This feels like the declaration of someone who has spent almost no time playing with these models or keeping up with AI over the last two years. Go look at the benchmarks and leaderboards for the last 18 months and tell me we're not progressing far beyond GPT4. Meanwhile models are also getting faster, cheaper, getting multi-modal capabilities, cheaper to train for a given capability, etc.

And of course there are diminishing returns, the latest public models are in the 90s on many of their benchmarks!

Re: Thoughts on a month with Devin

#159

Post by falcor84 »

toyetic wrote: post #131
falcor84 wrote: post #34

So for anyone who doubted SWE-BENCH's relevance's to typical tasks, it seems that its stated 13.86% almost exactly matches this 3 successes out of 20 pilot outcome. We're not quite there yet, but all of these issues seem to be solvable with current tech by applying additional training and/or "agentic" direction. I would now expect pretty much the textbook disruptive innovation process over the next decade or so, unti…

>> but all of these issues seem to be solvable with current tech by applying additional training and/or "agentic" direction. Can you explain why you think this. From what I gather from other comments it seems like if we continue on current trajectory at best you'd still need a dev who understands the projects context to work in tandem w/ the agent so the code doesn't devolve into slop.

> so the code doesn't devolve into slop

As I see it, this is pretty much a given across all codebases, with a natural tendency of all projects to become balls of mud if the developer(s) don't actively strive to address technical debt and continuously refactor to address the new needs. But having said that, my experience is that for a given task in an unfamiliar codebase, an AI agent is already better at maintaining consistency than a typical junior developer, or even a mid-level developer who recently joined the team. And when explicitly given the task of refactoring the codebase while keeping the tests passing, the AI agents are already very capable.

The biggest issue, which is what you may be alluding to, is that AI agents are currently very bad at recognizing the limits of their capabilities and continue trying an approach when a human dev would have long since given up and went to their lead to ask for help or for the task specification to be redefined. That's definitely an issue, but I don't see any fundamental technological limitation here, but rather something addressable via an engineering effort.

In general, I've seen so many benchmarks fall to AI in the recent decade (including SWE-BENCH), that now I'm quite confident that if a task being performed by humans can be defined with clear numerical goals, then it's achievable by AI.

And another way I'm looking at it is that for any specific knowledge work competency, it seems to already be much easier and time effective to train an AI to do well on it than to create a curriculum for humans to learn it and then to have every single human to go through it.

Re: Thoughts on a month with Devin

#160

Post by moffkalast »

epolanski wrote: post #70
moffkalast wrote: post #65

Earlier quoted context omitted.

That first one has always felt super crazy to me, I've figured out what lots of "arcane magic, don't touch" type of functions genuinely do since LLMs have become a thing. Even if it's slightly wrong it's usually at least in the right ballpark so it gives you a very good starting point to work from. Almost everything is explainable now.

I can relate, I have been genuinely amazed more than once by how it could "understand" some very complex code nobody dared to touch like you mention.

Kinda reminds me of that Glados quote, haha:

"These next tests require cooperation. Consequently, they have never been solved by a human. That's where you come in. You don't know pride, you don't know fear, you don't know anything. You'll be perfect."

It takes someone with no ego, no preconceptions, and infinite patience to delve in and come back alive.

Post reply on HN