Earlier quoted context omitted.
... What. It hallucinates and it doesn't compile, fine. It hallucinates and flips a 1 with a -1; oops that's a lot of lost revenue. But it compiled, right? It hallucinates, and in 4% of cases rejects a home loan when it shouldn't because of a convoluted set of nested conditions, only there is no one on staff that can explain the logic of why something is laid out the way it is and I mean, it works 96% of the time so…
As I said, you are still responsible for the quality control. You are supposed to notice that everyone is named Dave and tell ChatGPT to fix it. Write tests, read code, run & observe for odd behaviours. It's not an autonomous agent just yet.
Imagine you walk in 4 years down the track and try to examine AI generated logic committed under a dev's credentials. It's written in an odd, but certain way. There is no documentation. The original dev is MIA. You know there is something you defective from the helpdesk tickets coming through, but it's also a complex area. You want to go through a process of writing tests, refactoring, understanding, but to redeploy this is hard work. You talk to your manager. It's not everyone. Neither of you realize its only people named Dave/minority attribute X affected, because why would that matter? You need 40 hours of budget to begin to maybe fix this. Institutionally, this is not supportable because it's "only affecting 4% of users and that's not many". Close ticket, move on.
Only it's everyone named Dave. 100% of the people born to this earth with parents who named them Dave are, for absolutely no discernable reason, denied and oppressed.