Earlier quoted context omitted.
> Not sure if any of the test-driven development people have thought about this, but the same principle also applies there: if you debug and fix the code until it passes the tests, you can overfit the tests. It's no longer a good measure for code quality, once it has been explicitly optimized. You'd need new, previously unseen test cases. It would be an obvious mistake in machine learning to simply add failed test ex…
Yes but then the tests would formally describe the requirements - i.e.they would be a de-facto implementation. Congratulations, you just coded an (indirect) solution to your problem (a solution that you postulate is correct/bug-free)
Tests describing requirements acts as documentation, validation, and regression prevention with future refactors.