Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

Post reply on HN
Print view View original

51–60 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#51

Post by lhorie »

xiaq wrote: post #14
lhorie wrote: post #8

Earlier quoted context omitted.

test("when a metric becomes a target, it stops being a good metric", () => { runApp(); // look ma, lots of "coverage"! assert(true, 'No errors!'); }); // unfortunately paraphrased from real code

This test will inflate the test coverage, but it is a valid smoke test (assuming that any unhandled exception will cause the test to fail).

You shouldn't write tests like this; there's a high likelihood that the test will be flaky or not representative enough of production, and if the test fails, you often get completely non-actionable error messages.

If you just want to know that your app is broken, it's far better to monitor your live app (or staging environment or deployment pipeline or whatever) since that monitoring infrastructure can then be leveraged to collect other runtime health data in a more granular fashion.

Re: Coverage is not strongly correlated with test suite effectiveness

#52

Post by amw-zero »

You don’t even need to do a study to arrive at this conclusion. Has no one read Djikstra’s writings? https://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF

“Program testing can be used to show the presence of bugs, but never to show their absence!”

Everybody in our industry wants to bash math and say that anyone can write programs, but programs are logical systems and can only be fully understood with math and logic.

We have the tools for understanding and reasoning about infinitely large structures, programmers just refuse to use them, and even deride them.

Well, I am not in that camp, and I work on math and logic skills to help me develop working software. It isn’t perfect, but it’s the only path that makes sense to me.

> But complexity sells better and the market pulls in the opposite direction. I still remember finding a book on how to use "Wordperfect 5.0" of more than 850 pages, in fact a dozen pages more than my 1951 edition of Georg Joos, "Theoretical Physics"! It is time to unmask the computing community as a Secret Society for the Creation and Preservation of Artificial Complexity. And then we have the software engineers, who only mention formal methods in order to throw suspicion on them. In short, we should not expect too much support from the computing community at large. And from the mathematical community I have learned not to expect too much support either, as informality is the hallmark of the Mathematical Guild, whose members —like poor programmers— derive their intellectual excitement from not quite knowing what they are doing and prefer to be thrilled by the marvel of the human mind (in particular their own ones). For them, the Dream of Leibniz is a Nightmare. In summary, we are on our own. But that does not matter. In the next fifty years, Mathematics will emerge as The Art and Science of Effective Formal Reasoning, and we shall derive our intellectual excitement from learning How to Let the Symbols Do the Work.

Calculemus!

https://www.cs.utexas.edu/users/EWD/transcriptions/EWD12xx/E...

Re: Coverage is not strongly correlated with test suite effectiveness

#53

Post by void_mint »

tharkun__ wrote: post #41
_6pvr wrote: post #35

Earlier quoted context omitted.

> It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement. Tests are code. Code can be sloppy, fallible, useless. Writing a consumer before you write a provider doesn't make either more robust, just the point at which they meet more clear. You can certainly write a test that uses a function, but the test doe…

Very well said. I have nothing against TDD if that's something that helps a particular person write good code and good tests. I find though that people that write good tests are just people that write good tests. Most people write tests that assert on implementation details instead of inputs and output and that's easily doable via TDD as well.

Yes, I agree with this. Process exists to increase the bottom, but if you're trying to rely on TDD to get valuable tests, you're probably just thinking of the problem incorrectly (I think).

In my experience, the absolute best way to get valuable tests is to have someone whose job it is to own them, their correctness, and add to them when new issues arise. An SDET, or a rotating team member, or whatever. The average dev isn't going to write a test that's super valuable if they also wrote the code.

Re: Coverage is not strongly correlated with test suite effectiveness

#54

Post by sverhagen »

I've been asked many times what is the right code coverage percentage to aim for. I've also asked this many times in interviews, to tease out a discussion. My answer is: you need full coverage. I then continue to explain that I didn't say 100%. Although I believe that the average Java microservice (which is the general sphere that I move around) can easily achieve >98% coverage. Easily. But "full coverage" means that all reasonable scenarios should be covered. No code coverage tool is going to measure that for you (I think). Reasonable scenarios are... everything? Time is finite, so most teams will never be exhaustive, but one can try. The "80% coverage crowd" is throwing in the towel before even starting.

Re: Coverage is not strongly correlated with test suite effectiveness

#55

Post by kstrauser »

lwansbrough wrote: post #36

Now, I've never worked at a big company with lots of developers, but testing seems dramatically overvalued. My company currently employs zero testing (meaning zero automated tests). Anything that could negatively impact the company if it blew up is examined pretty closely and then set loose. Occasionally things break. Our company serves tens of millions of users monthly. We have bugs, we fix them as needed. We're not…

I don't like TDD. A lot of times I don't know what the code will look like until I write it, and whatever's on the screen is just a scratchpad for my thoughts until the final form becomes clear to me.

That said, I love having a good test suite. I updated a major internal project from Python 2 to 3 and my metric for being done was whether all the tests passed. When they did, we launched it in production, and it worked without serious bugs from the very beginning. I can't even imagine what that process would've looked like without some reassurance that I was catching at least most of the corner cases we'd previously identified.

Also, I love tests as a way of asserting that a bug or security issue stays fixed. In fact, I doubt it's possible to pass something like a SOC 2 audit without it.

Re: Coverage is not strongly correlated with test suite effectiveness

#57

Post by jeffbee »

lwansbrough wrote: post #48
jeffbee wrote: post #42

Earlier quoted context omitted.

I am a huge fan of unit testing and the best thing about TDD is getting your function signatures and other APIs sorted out before writing your implementation. Having to write the calling function first helps ensure that the called function has a signature that will be useful to other callers, instead of something awkward that seemed OK when you started writing the implementation but later turned out to be imperfect.…

The notion that you can get software right the first time seems... a bit naive to me, regardless of the methodology, respectfully. We solved the internal API problems in our app simply by either versioning internal services/APIs (ie. MyServiceClassV2), adding a new method signature and deprecating the old one, or updating the existing method signature, which is pretty safe in strictly typed languages. (And if you wri…

I find your comments quite interesting. How do you validate MyServiceClassV2 if you didn't bother writing any tests for MyServiceClass? One of the benefits of testing is it enables refactoring. Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything.

Re: Coverage is not strongly correlated with test suite effectiveness

#58

Post by spankalee »

thaumasiotes wrote: post #45
tharkun__ wrote: post #10

In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other peop…

> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input? How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input. Now, if your code is just pumping data from one point to anoth…

That's because you have such simple conditions. How do you compute them?

   if (x
really changes things. Did you test for x === y? for NaN? for very close doubles that should have passed/failed for business reasons? For types other than numbers if your language allows?

Expressions can have any number of edge cases that code coverage can't account for.

Re: Coverage is not strongly correlated with test suite effectiveness

#59

Post by kragen »

jacobsenscott wrote: post #20

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…

I think it was Brian Marick who pointed out that the great benefit of a coverage report is that it tells you what you forgot to think about when you wrote the test suite. One response to code that isn't covered is to write the tests to exercise that code, but there are a couple of other possibilities he suggests might be better:

1. Can the uncovered code be removed from the system entirely? Maybe if none of the tests for other parts of the system invoked it, it's not used at all. (This is less likely if you use a lot of test stubs.)

2. Maybe if you didn't think about the scenario where the uncovered code gets invoked when you were writing the test suite, there are other things you also didn't think about—and maybe some of them aren't covered in the implementation either. Write down the missed case and put the implementation away for a while—hopefully long enough to forget how it's implemented. Once you've forgotten, refer to your notes and write a suite of tests that covers the missed case as well as anything similar.

— ⁂ —

I agree that test-first improves your tests in the way you describe: your test suite is guaranteed to have nearly complete coverage. Also, you have some evidence that the test itself works rather than vacuously passing.

But I think there are two other benefits of test-first programming that are commonly undersold.

First, it makes programming more fun, because you have immediate feedback when you make a test pass.

Second, sometimes you write a test for code you haven't written yet, and you can see by looking at the test that your design sucks: you need seven objects and six method calls to do something simple, and one of the method calls has a boolean parameter, making the code incomprehensible. This feedback allows you to improve your design, possibly several times, before writing the implementation. This allows for faster design iteration than refactoring the implementation toward a better design does. I think this is what jeffbee is saying in https://news.ycombinator.com/item?id=28677978.

Re: Coverage is not strongly correlated with test suite effectiveness

#60

Post by falcolas »

amw-zero wrote: post #17
laanako08 wrote: post #3

This affirms my intuition that the power of test suites arises from their coverage of the data-cases, and call-sequence, and not simply from "visiting" more lines of code. This also is likely the underlying reason for the extreme effectiveness of fuzz-testing and property-based testing.

They go hand in hand. It’s obvious that missing branch coverage means your data cases are not exercising all of the edge cases in the code.

Not all code is reachable. Especially things like top-level try/except clauses.
Post reply on HN