Coverage is not strongly correlated with test suite effectiveness
neverworkintheory.org
Coverage is not strongly correlated with test suite effectiveness
1–10 of 178 posts
Re: Coverage is not strongly correlated with test suite effectiveness
#2In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness?
Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
Re: Coverage is not strongly correlated with test suite effectiveness
#3This affirms my intuition that the power of test suites arises from their coverage of the data-cases, and call-sequence, and not simply from "visiting" more lines of code. This also is likely the underlying reason for the extreme effectiveness of fuzz-testing and property-based testing.
Re: Coverage is not strongly correlated with test suite effectiveness
#4I usually just use Unit Test coverage as extra insurance that I didn't accidentally forget a path when writing up tests for new classes, or when trying to assess others tests to see if/what they miss. Outside of that, I basically ignore it.
That (per)mutation testing, sounds like pitest, which I've had fun with using to gauge the effectiveness of tests I've written in the past.
Re: Coverage is not strongly correlated with test suite effectiveness
#5In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
It depends. When you tell developers "you must have 100% code coverage" they usually write tests that don't actually validate any functionality and instead get into every if block. Tests are useful when they test edge cases and assert behavior.
I've told this story many times before but at a previous job a senior engineer told me "100% code coverage is useless and you shouldn't go for it" but since he was being dogmatic and not actually thinking about what he was saying he was arguing against something very sensible. I was testing an expert system where everything was large if/else trees encoded in types + configuration. I wanted to make sure I tested all edge cases and activated all of the blocks when they made sense.
I had to fight for that extra coverage and it was, in the end, a massive help.
Re: Coverage is not strongly correlated with test suite effectiveness
#6In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
If it takes you 3 days to write a test that covers a once-in-a-million condition and your service gets 2 requests per day, it will take you about 500,000 days to hit that condition once.
You've increased test coverage, but was it effective? Eh probably could do something more useful with your time.
(yes this is a contrived example, adjust numbers for your situation)
Re: Coverage is not strongly correlated with test suite effectiveness
#7This makes perfect sense: a simple function with two code paths that splits on the comparison of two signed integers immediately requires a minimum of three test cases for correctness, yet it only takes two to achieve 100% code coverage.
Checking for correctness for corner case values - maxint, minint, zero - adds a minimum of another 9 cases.
And it will take many, many more test cases if you're working with a weakly typed language and you're potentially comparing an integer with a floating point value. Or strings. And so forth.
Re: Coverage is not strongly correlated with test suite effectiveness
#8In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
test("when a metric becomes a target, it stops being a good metric", () => {
runApp(); // look ma, lots of "coverage"!
assert(true, 'No errors!');
}); // unfortunately paraphrased from real codeRe: Coverage is not strongly correlated with test suite effectiveness
#9In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
If it takes you 3 days to write a test that covers a once-in-a-million condition and your service gets 2 requests per day, it will take you about 500,000 days to hit that condition once. You've increased test coverage, but was it effective? Eh probably could do something more useful with your time. (yes this is a contrived example, adjust numbers for your situation)
Depends on what the consequence of that failure might be. It could be anything from completely unnoticed to ending somebody’s life, rarity is only one dimension of risk.
Re: Coverage is not strongly correlated with test suite effectiveness
#10 In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise
I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other people about this but to me it's a well known fact. There are so many ways this can go wrong.
I mean it's so easy to give the one counter example needed to break the myth of 100% test coverage being good for much: Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input?
More tests than is needed for "100% coverage" means that you actually executed some lines multiple times, hopefully with not just 10 happy path scenarios but with 1 happy path and 9 edge cases. Now remove the happy path scenarios for trivial code and also the edge case scenarios for trivial code and your coverage might only be 80% but you have the same actual test suite effectiveness. When you keep adding tests, add more to the 9 edge cases, staying with the same coverage but make the suite more robust.
Of course 20% is better than 0% and 50% is better than 20%. Somewhere between 50 and 100 is an optimal point. For lack of a good estimate, let's go with the old 80/20 rule. Something around 80% test coverage is what you can use to make a tool fail the build. If you pass the 80% mark you _might_ have a good test suite but it doesn't mean you stop there. It's just a reminder that you might have missed adding tests when that thing triggers a red build but don't use it to mean that you actually have a good test suite.