Earlier quoted context omitted.
Tests don't (can't) prove tthat code is correct. They are merely a rough plausibility check that the code isn't completely wrong and didn't regress. You generally can't derive the right code just from tests.
You can write tests about properties you care about which may not be everything. Generally in some of the more financial applications i've written I would be ok with people rewriting the app as long as it passes the tests. I've even written tests that say this set of input goes to this output, for various different subsets of input. Anything outside of the of the defined input sets fail validation. Than it randomly p…
A random sample of inputs that is hidden from the AI also won’t allow it to derive a corresponding implementation. And if the set of sample inputs is not hidden, then the AI is still free to produce an implementations that only works for those sample inputs.