> What are the numbers on how secure is human written code? We should have something to compare AI numbers to.
That's kind of what the article is about? Mythos is finding lots of security bugs in lots of human-written code. They can now compute some sort of baseline estimate of security bugs per N lines of human-written code or whatever. (Restricted to security bugs that the AI is currently capable of finding, but whatever.) Even before Mythos et al, we can look at historical security bug rates. We do have numbers for estimating the security of human written code.
> It seems more likely to me that you could spend $20 to find a vulnerability in a piece of software that costed you $20k in human labor.
Ok, but that's not what is being discussed in this subthread? The topic is whether or not we have data suggesting that AI-written code is or can be secure, and thus whether insecure human code is fated to replaced with secure AI code. I claim we do not have that data. Therefore, we don't have evidence to think that for the sake of security we should replace all human code with AI code, vs whether AI code is worse for security and so we should replace AI code with human code (that presumably has been vetted with AI, since we do have evidence for its effectiveness.)
If I were to guess, I would probably think that today's AIs are trained solely on mountains of insecure human code and so will probably produce more of the same. Tomorrow's AIs will have the benefit of being trained on human and AI code that has had a large swathe of vulnerabilities purged from it, and so they'll have a much better chance at writing secure code, at least.
It depends a lot on whether the failure modes of AI code generation lend themselves to exploitation as security vulnerabilities. (And whether they will continue to do so.)