GitHub Copilot loses an average of $20 per user per month

Earlier this year, we had this one developer who has worked with us for a while (8 months or so). He was never a top developer, but he was average.

Anyway, I have to review all of his code, and I've been doing this the whole time he worked for us. But, starting around the beginning of the year, I started to notice problem after problem coming through his merge requests (MRs). At first I gave the guy the benefit of the doubt, like I said this was unusual for him. He always needed moderate coaching, but the mistake ratio was way higher than usual.

Anyway, I did some basic coaching and whatever. I talked to him about the increase in errors and he said he was stressed with moving to a new house at the time. It made sense, so I decided not to even record the first performance review with HR.

But the problems kept coming and getting worse. It was getting to the point that he had so many problems, I wouldn't even address them all. I would only kick back the top dozen or so. I was noticing that his code had entirely different styling and voice across methods like it wasn't written by him. I also started to see weird lines that were trying to catch unusual edge cases that would never even apply in our scenario. There was one time where there was this insanely complex filter that fed into an insane regex that I couldn't decipher. I asked him about what it is even trying to do and he had no clue, he couldn't even explain the purpose of the line, let alone how it worked. I pushed him on where it came from and he said "StackOverflow". But I reversed searched it and couldn't find it.

We ended up doing performance reviews almost weekly for a while, and now I was formally writing them up. I told him that he was more of a burden on the company at this point. He was offering a negative value. I was getting stressed out because I was spending so long fixing his code and coaching him, that I was working longer hours because of him. Furthermore, I could have fired him and not even replaced him, just taken over his job myself and it would have been less work than this back and forth of trying to repair his work.

Eventually I did fire him. After he left and I talked to the team about him being let go, one of his friends told me that this guy had been using ChatGPT and Copilot for all of his work. It was a secret because our company doesn't allow AI tools in our codebase right now because of compliance reasons. Which is why he would rather say his mistakes were "copypasta from StackOverflow" as he always put it, than admit it was copilot.

That's not to say that CoPilot is bad. But its not ready to replace developers just yet. Even junior developers still need to wrangle and check the work coming out of the AI. And yes, people do notice. Even after this employee polishing the work coming out of the AI model, it still wouldn't work in our codebase.

And to the original point. Yes, these AI tools really are just passing increased burden onto the experienced developers. The juniors are using these tools as crutches to speed things up so they can watch more YouTube videos. But the code gatekeepers who have to defend the application are the ones with increased burden and workloads of fixing the mistakes the AI is generating. I've been saying for a long time that AI will simply separate the good from the bad in software development. For the past 15 years you could make $150k or more a year as a developer that is only capable of producing sub-par code. But those days are gone. The sub-par code is being produced by AI now. So you need to be at least mediocre now.

> If I need to traverse a tree or list or something, I'm letting AI write that code. Could I write it myself faster? No, and it's going to have an off-by-one error some non-zero portion of the time if I write it.

Many languages/companies have existing well understood solutions that _won't_ have errors. Maybe that is the disconnect? I can't remember the last non-interview time I had to write a non-trivial traversal.

To clarify, I have been paying the 10$ since it was released last year. I used it in neovim, vscode and found it helpful.

Cut to few months later with OpenAI API release, so many code completers released in many useful form factors willing to go wherever I code even if its Android Studio or Xcode. The product has stagnated completely.

I do expect it to change significantly in the future though, might go back then.

Maybe if you live around the bay area where you get paid absurd money by the hour for a job that can be done overseas for 10 bucks, outside or the rest of the world that 100 usd is per week tops, thats way too much i would pay for service that only barely helps my in my day to day...

If you need to constantly use chatgpt/copilot to do your job properly, maybe its time to learn something else because your job will be automated sooner or later, what will be more cheaper a 100/hr or the 20/month service.

I would hope developers value their time at least 100USD/hour. So unless the local option is less than 5 hours per year of maintenance, the subscription makes sense.