Live data from Hacker News

Devin: AI Software Engineer

cognition-labs.com

531–540 of 604 posts

Re: Devin: AI Software Engineer

#531

Earlier quoted context omitted.

Machines currently are at an amateur level, but amateurs across the board on the knowledge base. Amateurs at Python, Fortran, C, C++ and all programming languages. Amateurs at car engineering, airplane engineering, submarine engineering etc. Amateurs at human biology, animal biology, insect biology and so on. I don't know anyone who is an amateur at everything.

> Machines currently are at an amateur level, but amateurs across the board on the knowledge base. No, and that is a one of their limitations. LLMs are human-level or above on some tasks - basically on what they were trained to do - generating text, and (at least at some level) grokking what is necessary to do a good job of that. But, they are at idiot level on many other tasks (not to overuse the example, but I just…

Agree, totally valid points.

Also take a look at rare diseases and doctors [1], in which machines are already better at diagnosing thousands of different rare diseases. Is is fair to say that machines are better at diagnosing diseases in general, just because they diagnose rare diseases, which each one, every doctor will need to diagnose once or twice in his career? Not clear at all.

Right now we are constrained by data, but that constraint will go away in 5 years or so. Will AGI will be achieved by then effortlessly? I have my doubts. My sentiment is that, even if AGI is never achieved, every small advancement in reasoning ability, in context window, in multimodal sensors and actuators, will have a very broad effect on jobs, on the economy and the way we are currently producing anything.

[1]https://www.youtube.com/watch?v=_6GhUjblqKg&t=826s

Re: Devin: AI Software Engineer

#532
post #391

Surprised how calm and underwhelmed comments are. Sure it is no senior architect but the trajectory is insane. Wasn’t that long ago that LLMs barely managed coherent poems. Now it’s troubleshooting code problems on its own? Sure it’s just a gpt4 wrapper but that implies the same can be done with gpt5 and six etc. Project it forward and that does actually become non trivial

This assumes that AI will continue to improve at the same pace. GPT3 released November 2022 and yes, there have been staggering improvements since then.

My intuition is that we’re beginning to reach the bounds of what we can do with LLMs. All we’ve been doing is applying the technology to other mediums (images, video, code) and yeah we’ve been seeing some really awesome/terrifying results. It all looks like it’s 80% of the way there, from which the first thought is that in a couple of years it’ll be 100% of the way there and we’ll all be jobless.

This reminds me of the full self driving hype. The demos were so exciting and all that was left was getting these last couple edge cases worked out and then we’d all be in self-driving taxi pods….

Re: Devin: AI Software Engineer

#534
post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

It's worth pointing out that on their eval set for "issues resolved" they are getting 13.86%. While visually this looks impressive compared to the others, anything that only really works 13.86% of the time, when the verification of the work takes nearly as much time as the work would have anyway, isn't useful. The problem with this entire space is that we have VC hype for work that should ultimately still be being do…

as much as this response can seem comforting, i feel like its also very easy to under estimate just how quickly an ML model can learn. I'm genuinely concerned that in the next 10 years this will replace the average software engineer

Re: Devin: AI Software Engineer

#535

Earlier quoted context omitted.

There is not a limited amount of software engineering that can be done. There's only the amount of software engineering that it is _economical_ to do at any given point. If AI makes software development cheaper and more efficient, people will just use apply it to more use cases. It's never been the case that making programming cheaper has lead to fewer programmers -- quite the opposite. This change is roughly analogo…

The problem is that AI has the potential to turn a professionalized industry into a self-help one. Like, if I can pay $30 a month to have an AI make me several new apps a day, why would I need an app store? It's like fast fashion for programs. Low quality but dirt cheap.

Why I would need an app when most of the time I could just make it in Excel?

Re: Devin: AI Software Engineer

#536
post #249

Earlier quoted context omitted.

13% unassisted is crazy. That's probably half the performance of an intern that costs ~100k/year.

But how do you know if you are in the remaining 87%?

You have to check by hand, just like you have to check an intern's code.

Re: Devin: AI Software Engineer

#537
post #428

Let's get realistic here - I just beat GPT-4 at tic tac toe, since it failed to block my 2/3 complete winning line ... Sure, one day we'll have AGI, and one day AGI will replace many jobs that can be done in front of a computer. In the meantime, SOTA AI appears to be an airline chatbot that gets the company sued for lying to the customer. This is just basic question answering, and it can't even get that right. Would…

Well, I'll bring the perspective of a recent college graduate from India. Most students who started jobs in software aren't writing software for aeroplanes. They're writing crud apps, doing third-party API integrations or doing basic debugging. I worry that although not some specialized software devs but a lot will still have problems due to stuff like this. I'm not talking today, but say 2-3 years down the line, who…

Imagine in 2-3 years time how much more crud software will be built and will be there to support. I'd say at least an order of magnitude more.

Even if I would have an AI assistant if I work on something I still have to understand that piece of work, I have to understand the requirements to give correct orders to the AI. I might become more efficient but I don't see an order of magnitude speed up where now I work on 5-8 projects which are in one ecosystem and I don't see suddenly switching to working on 50-80 projects where each 10 of them are different ecosystem that I also have to understand to make valid decisions - even if these are crud apps.

Re: Devin: AI Software Engineer

#538
post #193

Earlier quoted context omitted.

If it was actually working for anyone, then they would be selling software engineering time at the same but slightly cheaper price as existing software engineering time costs today so they could capture those sweet margins. This is a company spending investor money selling pickaxe hand grips during a gold rush. For real evidence, look for companies selling engineering time much greater than the amount of their total…

It's hilarious. If Devin were any good, they wouldn't be selling access to it to random SWEs, they would be replacing Microsoft, Apple, Google, etc for that sweet sweet trillions of dollars! Where's the app they built in an afternoon using Devin? Where's the software product that Devin actually built a month ago and was being used by thousands of people? Their actual business seems to be closer to "Lets milk some of…

Yeah that's basically straight up the reasoning I came up after watching demos...

Re: Devin: AI Software Engineer

#539
post #73

As a developer but also product person, I keep trying to use AI to code for me. I keep failing, because of context length, because of shit output from the model, because of lack of any kind of architecture etc etc etc. I'm probably dumb as hell, because I just can't get it to do anything remotely useful, more than helping me with leetcode. Just yesterday I tried to feed it a simple HTML page to extract a selector, I…

I'll give you examples of how it helps me: 1) copilot is a terrific auto complete, and writes tremendous amounts of repetitive boilerplate 2) copilot can help me kickstart writing some complex functions starting from a comment where I tell it what is the input and expected output. Is the implementation always perfect or bug free? No. But in general I just need to review and check rather than come up with the instruct…

> 1) copilot is a terrific auto complete, and writes tremendous amounts of repetitive boilerplate

I term this "low-entropy code". Copilot is great at writing heaps and heaps of low-entropy code.

The thing is, if you're not paid by LOC, and care about your system as a whole, you normally strive to get rid of code if possible (any code is liability), and make the rest of it high-entropy.

Today's terrific autocomplete is tomorrow's legacy shit you have to deal with.

Re: Devin: AI Software Engineer

#540
The lack of attention this development is getting on here is astounding. Other developers I am talking to about it brush it off, then change topic.

So many comments about how insufficient the tool is.

Our heads are really in the sand, I'm afraid.

Post reply on HN