Coverage is not strongly correlated with test suite effectiveness

It tends to be better for broad concerns which trigger somewhat rarely, tend not to be otherwise caught, but can be quite concerning. Crater is a good example of that, it’s infeasible to build & test every public crate on every CI check, but it’s a useful sanity check when release arrives or when messing with the more subtle aspects of the compiler.

Pototyping to define the problem or API is fine, but most people don't have the discipline to tear it out and start over when they finally do have a well defined problem.

It is somewhat unfathomable to me that you can have an idea if what production code to write, but no idea what test to write. Certainly you can "expect this page has a button" if you are about to add a button and write the test first. Or "expect this method to add a record to the database", etc. Certainly you are about to write code that does something, so just write a test to expect that thing to happen.

Especially in the case where someone is changing their mind all the time, you are dead in the water without tests to support your changes. You need to manually re-test everything every time they change their mind.

Testing after the fact is not that useful for catching regressions because it is unlikely enough tests will be written due to pressure to release. Also a much greater percentage of the tests written after the production code tend to be vacuous tests.

1. We need some way to compare # of tests across differently-sized codebases, i.e a percentage metric like coverage 2. The fact that correlation disappeared when controlling for absolute # of tests seems to suggest it's still encouraging people to write more tests. Once your org gets to a certain size you need these blunt instruments like minimum coverage requirements because you can't just rely on individual excellence anymore.

The different behaviors on this space of points are due to the control flow and mathematical expressions within the function. You could thus model the function's behavior as a partition of the input: the different sets of inputs that will behave "identically" WRT the control flow inside the function. You could probably formalize this with a suitable notion of continuity, but I haven't bothered.

Accordingly, testing corresponds to searching this space for problematic partitions. It's a bit like playing battleship: if you know a lot about the possible shapes of errors in this mathematical space, then you can choose a few small points that can totally prove their absence.

Even if you don't know the shapes though, by choosing your points carefully, you can rule out shapes that are "sufficiently large". This is the impetus behind boundary testing; you don't prove that errors don't exist, but you can prove that they are relatively "pointlike"; that there aren't arbitrarily wide swaths of the input space that all exhibit the same erroneous behavior. Since the input space is usually vastly multidimensional, there's usually pretty hard limits on how much you can actually reduce the set of possible "problem shapes," though.

A lot of in-practice software testing out there (I can only really speak about the work I've seen at Google) is statistical in nature: you run a huge amount of tee-d or saved production traffic through a service, and run a massive diff of the output, after painstakingly making your service deterministic. By doing this, you can show that a "typical" query is unlikely to cause a "catastrophic" failure.