Debunking Devin: "First AI Software Engineer" Upwork Lie Exposed [video]

An extremely solid and convincing rebuttal. Sad. I wonder what the Devin team will say in response, if anything. Summarizing the video:

• Devin is sold as being able to solve arbitrary Upwork tasks. In the video demo the problem it was asked to solve doesn't match the stated requirements of the customer (who asked for setup instructions, not code).

• Devin is shown fixing errors in the source of a GitHub repo, but the files it's shown editing don't actually exist in that repo and some of the errors its fixing are nonsensical, of the type that'd never be made by a human. Inference: Devin must be fixing bugs in files it has itself created, but that's not clearly indicated.

• There is no need to do any coding in the first place, because the README in the repository has all the instructions needed to achieve the task ready to go and they still work fine with only a one-line tweak, even though the repository is old. This is why the customer asked for instructions for how to run it on EC2 rather than for some coding. Devin didn't seem to read the README or understand that it only had to execute a couple of pre-existing Python scripts. The output in the video makes it look like the task was complex and sophisticated, with a long plan and many check boxes showing work completed, but the work was in fact pointless and redundant.

• Devin's code changes are bad, e.g. writing its own low level file read loop instead of using the standard library properly.

• Although the video makes it look like Devin did the task quickly, and the video creator was able to do the requested task in ~30 minutes, the timestamps in the chat show the task stretching over many hours and even into the next day.

• Devin does nonsensical shell commands like `head -n 5 foo | tail -n 5`

The strange mistakes lead to questions about what underlying model it's using. I don't think GPT-4 would make mistakes like that.

The Internet of Bugs guy is an AI fan and uses coding AI himself, but points out that the company behind it says you can "watch Devin get paid for doing work" which isn't actually supported by their video evidence when watched carefully.

I like hearing this balanced opinion. Generative AI is awesome, but demos around it should be honest and transparent. Looking at you Google as well. I don't know if I can trust a single Google demo for a while. They must do 0 edit, cut demos for quite some time in order to build that trust. Also this fake happy, go lucky, childlike communication after the edits/cuts is cringey. Unless you are doing it real-time like OpenAI did.

Based on the slogans said around Devin I decided to ignore it completely - so while I couldn't say it's bs for sure, I did feel the slogans are embellished and too good to be true.

Also for some reason I don't like the name for it at all. I don't understand how it could be so poorly chosen. Not that names can always give insight, but this name somehow was so off putting to me.

it is confirmed using gpt-4 though (not sure which version)

Honestly, I don't fully agree, at least for now. AI writes too much code that isn't optimized until you go in and ask it to edit a certain block in a certain way. At least that's been my experience so far.

So, the cost of needing to go through all the written code afterward and do a ton of code reviews/edits is more expensive than giving a good engineer a good AI.

I'm sure though that in a year or two we'll just be doing reviews, and edits will be rare...

There's still the cost of reviewing bad code. If the task was "completed" but you have to spend 20 mins looking over the code or the whole 1-hour-long process to "understand" what it did, it still failed.