Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

Post reply on HN
Print view View original

111–120 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#111

Post by Supermancho »

throwawaylinux wrote: post #23
falcolas wrote: post #7

This makes perfect sense: a simple function with two code paths that splits on the comparison of two signed integers immediately requires a minimum of three test cases for correctness, yet it only takes two to achieve 100% code coverage. Checking for correctness for corner case values - maxint, minint, zero - adds a minimum of another 9 cases. And it will take many, many more test cases if you're working with a weakl…

Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are…

I wouldn't even write this way because the else confuses what's happening.

    void fn(int a, int b) 
    {
        string ret = "not";

        if (a == b)
            ret = "equal";

        printf(ret);
    }
*I understand, this is not a good optimization in every language.

Re: Coverage is not strongly correlated with test suite effectiveness

#112

Post by malthuswaswrong »

arwhatever wrote: post #81

High coverage does not necessarily imply test suite effectiveness, nor does it even guarantee that your test suite provides positive value. But 0% coverage does tell you a lot about your test suite effectiveness. Also, beware the person too eager to proclaim that high test coverage is not necessarily good! Last time someone told me this was discussing process maturity during a job interview. I let the team slide with…

That's a shame. You could have gained new experiences and taught a team something new.

Writing a program is easy and done a million times a day. Fixing a department is hard and people rarely get an opportunity to do it.

Re: Coverage is not strongly correlated with test suite effectiveness

#113

Post by AlexCoventry »

saalweachter wrote: post #65
sverhagen wrote: post #54

I've been asked many times what is the right code coverage percentage to aim for. I've also asked this many times in interviews, to tease out a discussion. My answer is: you need full coverage. I then continue to explain that I didn't say 100%. Although I believe that the average Java microservice (which is the general sphere that I move around) can easily achieve >98% coverage. Easily. But "full coverage" means that…

I've occasionally been lulled into a false sense of security by the difference between "every line is run" and "the function is fully tested". On the other hand, even if I'm half-assing the unit tests, and implementing the bare minimum necessary to reach whatever arbitrary coverage% is being sought ... I still find bugs, bugs that almost certainly would have a real impact in production, even if I already ran an end-t…

That's my experience, too. Coverage isn't a good proxy for test quality, for sure, but it's a good proxy for having looked carefully at every line. I think coverage testing is useful as long as you're prepared to throw those tests away when you refactor, or at least ignore them during refactoring.

Re: Coverage is not strongly correlated with test suite effectiveness

#115

Post by exdsq »

gilbetron wrote: post #75

Say we have an application covered by 1000 tests. We fast forward a few years and (amuse this crazy notion) the application is decommissioned. We look and see that 500 tests never failed; those 500 always passed. Did we waste dev time by writing those tests? It's an interesting question to think about.

If I have a phone I need for work and insure it because if it breaks I could lose my client, but never actually need to claim on it, did I waste my money?

Re: Coverage is not strongly correlated with test suite effectiveness

#116

Post by SilverRed »

jacobsenscott wrote: post #20

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…

My favorite peer review trick is reverting the code change and showing that the test still passes. No tests will cover everything but its not hard to apply some simple tricks that raise the bar of actual coverage.

Re: Coverage is not strongly correlated with test suite effectiveness

#117

Post by exdsq »

lwansbrough wrote: post #66
jeffbee wrote: post #57

Earlier quoted context omitted.

I find your comments quite interesting. How do you validate MyServiceClassV2 if you didn't bother writing any tests for MyServiceClass? One of the benefits of testing is it enables refactoring. Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything.

It's not really about validation. Either MyServiceClass works or it doesn't (whether that's DX, or some other problem), and if it doesn't you make a new one or fix it. Most code isn't so complicated that coming up with the correct method signature is prohibitively difficult without writing tests. If down the line the way it was written isn't working, maybe you need a V2. Anything important that needs updating to V2 g…

It really depends on what your building, how quickly you need to build it, and what the client cares about. It could be no-one cares if MyServiceClass has a small bug that causes it to crash and restart from time to time - I worked in a startup that had something like that and it was fine within the needs of the company. Nowadays I work on financial stuff and a crash could cost ££££ so it's unacceptable, meaning we need more testing to check stuff.

Also with interfaces, they matter more if your working in a large distributed team. Sometimes you need to have the method signatures designed upfront so other teams can work on it, meaning it's less trivial than say an internal subroutine used in a private class.

Re: Coverage is not strongly correlated with test suite effectiveness

#118

Post by optymizer »

jacobsenscott wrote: post #20

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…

I found TDD useful for a well defined problem or an agreed-upon API. For apps for example, especially those not well defined and designed as-you-go, where the designer and PM might change their minds frequently after toying around with the app or getting user feedback, TDD is a lot of overhead and tests after writing the code are primarily useful for preventing regressions when somebody else changes your code.

Re: Coverage is not strongly correlated with test suite effectiveness

#119

Post by SilverRed »

KZeillmann wrote: post #29

A team getting to 100% test coverage and enforcing it feels like an application of Goodhart's Law. When a measure becomes a target, it ceases to be a good measure. At my last job, we required 100% code coverage for most code, and the other parts of the code (in ideality) were marked with ignores that were well-thought-through. In practice, I ended up writing a bunch of test cases only to hit the if blocks. It didn't…

I have never used the % stat as anything more than curiosity but I have certainly looked at the highlighted reports generated to see large blocks of untested code which is genuinely useful info to see.

Re: Coverage is not strongly correlated with test suite effectiveness

#120

Post by SilverRed »

dehrmann wrote: post #92

I find it somewhat rare that I update code and replace an implementation in a way that tests immediately pass. That said, I think the value of unit tests isn't so much in the coverage metric as much as 1) showing someone did the legwork to test the code 2) document some amount of caveats and expected behavior, and 3) provide the next person to edit the implementation a small test "framework" (mocks, dummy data) to bu…

The tests provide a useful confirmation step. You make a change and the tests go "Hey did you know that this affected the output in these 3 places?" and most of the time you did know that and update the tests, but often you see a change you didn't actually expect and can now look deeper in to it.

I have also found immense value in tests that just check that literally nothing has changed. Like one that checked that the columns of a db table have not changed and when they have, the test reminds you that you need to decide if the new column is blacklisted from a certain feature and then add the column name to the test so it passes again.

Post reply on HN