This is the interesting bit
> "Why did it pass reviews? Well, because of deadlines. And because there is simply so much code (and so much unparsable/misleading documentation) that it's simply impossible to review all of this. And because things move so fast that nobody understands the CI pipeline anymore, and the explanations of the agent are convincing enough that surely, it knows better than you?"
I have come to realize that AI is so "successful" because the system in which it is being deployed was designed to push product as fast and cheaply as possible from the start.
Humans are usually overworked and stretched to their breaking point, which I originally saw as the source of our broken software woes, which, like our streets in the US, just get a new layer of asphalt to cover up the crumbling bits each year instead of rebuilding the infrastructure with reliability and longevity in mind.
My former employer was using both Claude and Codex for firmware that was driving an over-burdened power circuit that itself was partially designed with ChatGPT. All of the individuals involved approach LLMs with god-fearing reverance because they do not understand _how_ the LLM works, just that it _does_ in a "good enough" way and they can offload their thinking, which is something we all wish we could do because thinking is hard, time-consuming and costly. I get it.
But like you mentioned, tests were being passed, not because the code was sound, but because the tests were altered to match the results. This is not necessarily the fault of the agent, either; it's just interpretting the prompt(s) - written by a flawed human, btw - with stochastic mechinations that seem to make a great deal of sense on the surface, but remain unable to be followed or repeated by the brains of (most of) its users.
As a rresult, I had to deal with product that work great in the field...at least at first, before it start literally catching fire, ruining its own powertrain because everything the agents touched became too complex with too many subtle cracks in the veneer to review properly. The system (read; capitalism) demanded viable product quickly to please investors, and the burnt-out humans who decided to try this AI thing ended up trusting it nearly completely, so any ideas of repeatable and complete testing, diagnostics and root cause failure analysis morphed into a sloppy "it works on the bench" checklist before being sold to a customer who had come to trust that their deceptively simple product would just work as advertised.
I'm going to die on the hill that AI as a replacement for our brains is precisely how we will make ourselves go extict, but I am old enough to already be regarded as a crufty dinosaur who is stuck in his ways, and I'm made peace with all of that. What I can't get my head around is watching people use this awesome tool (and it is, admittedly, awesome) to literally just speed up all the mistakes they were already making. Perhaps it is because I am aging, but slowing down and having a think seems more valuable to me now than it ever has, especially when creating something new. AI is powerful and, like any good tool, could be useful in the right hands, but more often than not I see it being used as an accelerant for all the worst parts of product development to appease a market that has suddenly been told they can now pick all three points on the Iron Triangle instead of just two. This makes about as much sense to me as taking a laxitive when you already are suffering diarrhea.