Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

81–90 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#81
High coverage does not necessarily imply test suite effectiveness, nor does it even guarantee that your test suite provides positive value.

But 0% coverage does tell you a lot about your test suite effectiveness.

Also, beware the person too eager to proclaim that high test coverage is not necessarily good!

Last time someone told me this was discussing process maturity during a job interview. I let the team slide with this answer. Found out their coverage was effectively 0 and their process maturity was "break everything until right before each quarterly release" and I was out of there before my first week on the job.

Re: Coverage is not strongly correlated with test suite effectiveness

#82

Earlier quoted context omitted.

Let's say I mistype 'a == b' as 'a Let's add 1 to a before the equality comparison in order to meet a new business requirement. fn(1, 1) still works, but does fn(maxint, maxint)? Or does it suddenly throw an exception (or worse, silently roll over to minint)?

Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'. I'm not saying more tests can't catch specific bugs you might come up with, I'm asking how you can choose numbers that have fundamental edge cases for this specific requirement without looking at the actual implementation. I don't think you really can. maxint/minint/0/-1/1 are generally commo…

Now you understand - testing is extremely close to useless. You can't perfectly choose the test cases for code without looking at its implementation. And by that point, you are just testing the implementation and working backwards.

What you want is a specification of the behavior, and to verify that the implementation satisfies the specification. This will be proven in the general case, and then you don't need to think about individual cases.

Re: Coverage is not strongly correlated with test suite effectiveness

#83

Earlier quoted context omitted.

Is it reasonable? It depends on your goals. Do you want to ensure that, for the set of well known edge cases, your function is correct? Proving the function is correct for the known edge cases for integers and integer comparison is not excessively hard. Let's go for the worst case of edge cases, and: - Pick the input pairs that are problems with ints: the set of (minint, maxint). - Pick the input pairs that are probl…

I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", that is a fallacy. If you want that, you have to use formal. The problem is not this one particular function, it's transferring this methodology to something more complex. A reasonable set of tests that does not aim to prove correctness is quite entitled to ass…

You're testing for the characteristics of your function - inputs and their outputs. If your function takes strings, there's a well known set of strings that are typically known to cause issues. It's the same for every data type. And for every operation against/between those data types, there's a known set of issues.

You can be pretty certain of a function's correctness (or more accurately, the correctness of a refactor - what unit tests are most useful at, in my opinion) without having to re-write the function in a slightly different way.

But to get back to the topic at hand - code coverage is a pretty bad measure of program correctness. What I was really trying to show is how bad code coverage is at predicting correctness for even the shortest and simplest of functions.

Re: Coverage is not strongly correlated with test suite effectiveness

#84
post #72
post #69

Aiming for 100% test coverage actually produces negative value. You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage for a few months and you will see it for yourself. The negative value comes from: 1. The time wasted on writing all these tests that are mostly ceremonious in nature. But, more importantly: 2. It makes refactoring a big pain in the ass. Why? Because 100% test c…

It is very unpopular view. I have never been a fan of unit testing and instead I prefer end to end functional tests -- where you write tests that verify your application still behaves exactly as expected but does not care how it is implemented. This usually requires much less code, does not deter refactoring and also focuses on the one thing that is really important for the client. Since the outside interface of the…

> Also my personal style is to do shit ton of refactoring. This means I usually start with something that only vaguely resembles the end result. I spend a lot of time moving stuff around until I get rid of everything that I don't like.

Same here.

I do think there's value in some unit testing: mostly for small units whose behavior is a bit tricky to verify by just looking at the code or running the application. These are usually functions with tricky mathematical expressions, or something like that.

Re: Coverage is not strongly correlated with test suite effectiveness

#85
Beyond a certain point, test coverage has diminishing returns.

You end up testing failure cases which are better addressed by good error handling. For example, inability to open a file due to file permissions can be covered in a more general way.

Erlang is one of the most effective platforms for making reliable software. It has the concept of "supervisors" which will monitor and restart code if there is a problem. This handles both known problems which are appropriately managed by the standard strategy as well as unexpected problems.

You can also use other static code analysis tools that ensure that you have dealt with every possible return code from a function.

Other things become more useful than testing, e.g. implementing observability to tell you what is happening at runtime.

Instead of more unit tests, I would prefer to have more validation checks when deploying code and in production. A validation check for Blue/Green deployment prevents bad code from going live. A production check ensures that nothing bad is happening that affects users, e.g. if we would normally get 10 signups per hour, and we are now getting zero, then it's time to page someone. I will take that over an extra 10% code coverage any day.

Put the effort that you were going to use on code coverage into something that gets higher value.

Re: Coverage is not strongly correlated with test suite effectiveness

#86

Earlier quoted context omitted.

> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input? How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input. Now, if your code is just pumping data from one point to anoth…

That's because you have such simple conditions. How do you compute them? if (x really changes things. Did you test for x === y? for NaN? for very close doubles that should have passed/failed for business reasons? For types other than numbers if your language allows? Expressions can have any number of edge cases that code coverage can't account for.

Depends on the language/type safety offered. If your language has a NaN (javascript) and you can assert the expected behavior of x < NaN (false) or if you don't even want to encounter that, then you're able to reason back to the generation of the inputs. Wherever x and y came from, if there was a possibility of an NaN being a value, you want the generating function to exception or cast via some sort of validation before handing them off to other unsuspecting functions.

Re: Coverage is not strongly correlated with test suite effectiveness

#87
post #69

Aiming for 100% test coverage actually produces negative value. You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage for a few months and you will see it for yourself. The negative value comes from: 1. The time wasted on writing all these tests that are mostly ceremonious in nature. But, more importantly: 2. It makes refactoring a big pain in the ass. Why? Because 100% test c…

3. Your code base may start to become contorted. I've seen good programmers create bogus classes to allow test-time mocking, or add oddball env vars and configurations to let the test harness manually reach every last line. Even if that line is not worth testing:

    if(!(x=malloc(BUF_SIZ)) || ENV[TEST_MEM_FAIL_12]) {
        exit(1);
    }
Tying code and tests this tightly discourages refactoring.

Another example: a different code base that I helped on had some insanely long one-line conditionals that anyone would break into multiple lines ... but doing that would have blown the "100%" coverage.

Re: Coverage is not strongly correlated with test suite effectiveness

#88

Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little abo…

Possibly the most valuable post I've read on these news pages.

Re: Coverage is not strongly correlated with test suite effectiveness

#89
post #72
post #69

Aiming for 100% test coverage actually produces negative value. You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage for a few months and you will see it for yourself. The negative value comes from: 1. The time wasted on writing all these tests that are mostly ceremonious in nature. But, more importantly: 2. It makes refactoring a big pain in the ass. Why? Because 100% test c…

It is very unpopular view. I have never been a fan of unit testing and instead I prefer end to end functional tests -- where you write tests that verify your application still behaves exactly as expected but does not care how it is implemented. This usually requires much less code, does not deter refactoring and also focuses on the one thing that is really important for the client. Since the outside interface of the…

So true. Testing and documentation "freeze" a solution. Going too deep with them from the beginning has a similar effect to early optimization. There is an economic factor in which code is mature enough to invest in testing/optimization/documentation, before that point effort is wasted.

Re: Coverage is not strongly correlated with test suite effectiveness

#90

Earlier quoted context omitted.

I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", that is a fallacy. If you want that, you have to use formal. The problem is not this one particular function, it's transferring this methodology to something more complex. A reasonable set of tests that does not aim to prove correctness is quite entitled to ass…

You're testing for the characteristics of your function - inputs and their outputs. If your function takes strings, there's a well known set of strings that are typically known to cause issues. It's the same for every data type. And for every operation against/between those data types, there's a known set of issues. You can be pretty certain of a function's correctness (or more accurately, the correctness of a refact…

I know that's the idea, what I haven't seen is any real data showing that is a significant improvement in testing. You pick a few things, there are a lot more that can go wrong, off-by-one can come up in comparisons easily, shifting left or right can happen easily, assortment of bitwise operations can happen. And that's just integers. When you get into floats what are you going to plug in all common math constants as well as a range of values around each of those, various products and summations and factors of those things? No of course not. It's perfectly reasonable to simplify the test based on what the function is supposed to do.

I'm not saying don't test any integer corner cases, I'm asking for the reason why the original poster says including those is good (and listing only 9 of possible 25 combinations to boot). In any case, why is that set of tests the golden one? That was just asserted as fact without any real reasoning I could see.

If you tell me those 25 are the right set of tests, I can say you're wrong because 2 is also very common in computing so 2 and possibly -2 should be tested as well. My 49 tests are clearly superior. Then someone comes along and says well 10 is common in scientific or business logic, so better add 10 in there as well. Using the same arguments as you did to reach 25 we can now get to 81. And so on. Apply it to a function with 5 arguments and we're up to 60 thousand tests. Not reasonable.

Post reply on HN