Devin: AI Software Engineer

Let's get realistic here - I just beat GPT-4 at tic tac toe, since it failed to block my 2/3 complete winning line ...

Sure, one day we'll have AGI, and one day AGI will replace many jobs that can be done in front of a computer.

In the meantime, SOTA AI appears to be an airline chatbot that gets the company sued for lying to the customer. This is just basic question answering, and it can't even get that right. Would you trust it to write the autopilot code to fly the airplane? Maybe to write a tiny bit of it - just code up one function, perhaps?

I sure as hell wouldn't, and when it can be trusted to write one function that meets requirements and has no bugs, it's still going to be a LONG way before it can replace the job of the developers who were given a task of "write us an autopilot".

Yeah I'll only give it tasks where it needs to spot patterns and do something obvious, and even then I'll check it make sure it hasn't just omitted random stuff just for shits and giggles.

TBH I'm more surprised when I don't need to help it now. After about 3 times where it cycles between incorrect attempts I just do the job myself.

I disabled copilot since it consistently breaks my flow.

I bought a pair of 350 dollar boots that can be repaired many times.

That was five years ago, with a recent re-sole this past summer.

This isn't an exaggeration by the way, but the cobbler who did the resole thanked me for bringing him the job, as it was a genuinely enjoyable experience for him. I assume most of his work is repairing suitcases going by the other clientele in the shop.

But sure, exaggerate that to be an artisan you need to be selling 1000 dollar shirts, rather than doing a 100 dollar service that doubles or triples the life of a decently made item.

We're not that far from a major turning point.

Currently these models don't provide an adequate enough confidence measure that prevents them from maximizing their potential. In the next few years we're going to reach a point where models will be able to tell if something is possible and avoid hallucinating, guaranteeing much better correctness. Something like that would be absolutely killer.

If you add on a top-down approach using a framework, such that it can architect a system down into small individual components, then that's a recipe for a really great workflow. The models we have now really shine in doing automated unit tests, and small bits of code to avoid limits with context size. Making the interfaces obvious enough, and being able to glue things together using obvious connections seems very possible.

I really do think that in the next few years we're going to see one of these tools really do well.

Yeah seems like this could be replicated by another AI dev fairly easily.