function (int i) {
return 1 / i;
}
Coverage is not strongly correlated with test suite effectiveness
11–20 of 178 posts
Re: Coverage is not strongly correlated with test suite effectiveness
#12In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
Coverage only indicates which parts of the codebase were touched by the test suite. A big test suite size doesn't mean high coverage.
Coverage can be increased without increasing the test suite by reducing the code base size (within pratical limits obviously)
Personally, I only find coverage as a good indicator of which code still needs to be tested, like forgetting some edge cases or conditional branches.
Re: Coverage is not strongly correlated with test suite effectiveness
#13In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
Quality of trests is important, if you’re chasing 100% you’re liable to be writing inconsequential or incorrect tests just to achieve a number.
Re: Coverage is not strongly correlated with test suite effectiveness
#14In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
test("when a metric becomes a target, it stops being a good metric", () => { runApp(); // look ma, lots of "coverage"! assert(true, 'No errors!'); }); // unfortunately paraphrased from real code
This test will inflate the test coverage, but it is a valid smoke test (assuming that any unhandled exception will cause the test to fail).
Re: Coverage is not strongly correlated with test suite effectiveness
#15"(F)ive large open source Java projects" is hardly a big and diverse enough sample to be making such a sweeping conclusion.
Re: Coverage is not strongly correlated with test suite effectiveness
#16In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
The correlation obviously exists for low levels of coverage.
Above 80% or 90% it becomes a poor measure.
Re: Coverage is not strongly correlated with test suite effectiveness
#17This affirms my intuition that the power of test suites arises from their coverage of the data-cases, and call-sequence, and not simply from "visiting" more lines of code. This also is likely the underlying reason for the extreme effectiveness of fuzz-testing and property-based testing.
They go hand in hand. It’s obvious that missing branch coverage means your data cases are not exercising all of the edge cases in the code.
Re: Coverage is not strongly correlated with test suite effectiveness
#18function (int i) { return 1 / i; } 100% coverage means nothing here when i = 0.
Furthermore, a line of code is arbitrary, and so is this metric.
Re: Coverage is not strongly correlated with test suite effectiveness
#19In practise, doesn't increasing the coverage highly correlate with increasing the test suite size, therefore proving the effectiveness? Conversely, I struggle to think how coverage could be increased significantly without increasing the test suite size in reality.
It depends. When you tell developers "you must have 100% code coverage" they usually write tests that don't actually validate any functionality and instead get into every if block. Tests are useful when they test edge cases and assert behavior. I've told this story many times before but at a previous job a senior engineer told me "100% code coverage is useless and you shouldn't go for it" but since he was being dogma…
100% code coverage is still not a good use of everyone's time in most projects and languages. There's a bunch of trivial code that even feels wrong to test. Spend more time cooking up edge case tests that execute some of the branches way more than once.
Think of your code like a heat map. Higher heat on lines that get exercised more often by your tests. It's fine that _some_ code has no color at all, while you some other paths to be bright red in the end, instead of always just going for a uniform orange for everyhing.
Re: Coverage is not strongly correlated with test suite effectiveness
#20It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful.
Test first also improves the quality of the tests just because it forces you to write a failing test first, and then write the code to make it pass, and you do this over and over as you build up the production code. In this way you are actually proving that the code you are adding is making tests pass.
That doesn't mean your tests are great - you are probably still mostly testing the happy path. But then you have a solid foundation to layer mutation/fuzz testing on.
I can't tell you how often I see people write tests after writing the production code - and I can go and delete most of the production code and the tests still pass. Writing quality tests for existing code is much harder than writing tests for code that doesn't exist - it is counterintuitive if you haven't done it both ways.