Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

Post reply on HN
Print view View original

21–30 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#21

Post by gravypod »

tharkun__ wrote: post #19
gravypod wrote: post #5

Earlier quoted context omitted.

It depends. When you tell developers "you must have 100% code coverage" they usually write tests that don't actually validate any functionality and instead get into every if block. Tests are useful when they test edge cases and assert behavior. I've told this story many times before but at a previous job a senior engineer told me "100% code coverage is useless and you shouldn't go for it" but since he was being dogma…

100% code coverage is still not a good use of everyone's time in most projects and languages. There's a bunch of trivial code that even feels wrong to test. Spend more time cooking up edge case tests that execute some of the branches way more than once. Think of your code like a heat map. Higher heat on lines that get exercised more often by your tests. It's fine that _some_ code has no color at all, while you some o…

> 100% code coverage is still not a good use of everyone's time in most projects and languages

The key here is most. Think about your use case before applying anything you read online.

> It's fine that _some_ code has no color at all, while you some other paths to be bright red in the end, instead of always just going for a uniform orange for everything.

This was the logic tree of a device used for health care work. Cost of failure was high. I had very good coverage of all edge cases that other systems could produce and tested a lot of extremes + a DSL for describing the input state.

The important thing here: an expert system is not most programs and the cost of a failure should really drive what your testing methodology is.

Re: Coverage is not strongly correlated with test suite effectiveness

#22

Post by kazinator »

Well, the number of case combinations that can occur can be exponential in the code size, whereas coverage is linear.

Say we have a sequence of five one-branch if statements: if (cond_i) { do_stuff_i } for i from 0 to 4. We can get complete coverage by executing that sequence just once with all conditions being true. But there are 32 distinct pathways through it, which could have all sorts of bugs.

Coverage also doesn't test for correctness at all; suppose that the one case that hits 100% coverage by executing every do_stuff_i produces the wrong result for that case. 100% coverage was still achieved.

A test suite consisting of nothing but a coverage test is ineffective in doing anything other than showing that the software doesn't bomb or infinite loop.

Re: Coverage is not strongly correlated with test suite effectiveness

#23

Post by throwawaylinux »

falcolas wrote: post #7

This makes perfect sense: a simple function with two code paths that splits on the comparison of two signed integers immediately requires a minimum of three test cases for correctness, yet it only takes two to achieve 100% code coverage. Checking for correctness for corner case values - maxint, minint, zero - adds a minimum of another 9 cases. And it will take many, many more test cases if you're working with a weakl…

Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this:

  void fn(int a, int b)
  {
    if (a == b)
      printf("equal");
    else
      printf("not);
  }
What are the 3 test cases you would write? What are the 9?

  fn(1, 1) -> "equal"
  fn(1, 0) -> "not"
What more useful tests are there here? Other than just picking arbitrary numbers, maxint/minint/1/0/-1/etc are not significant edge cases here.

Re: Coverage is not strongly correlated with test suite effectiveness

#25

Post by amw-zero »

windows2020 wrote: post #11

function (int i) { return 1 / i; } 100% coverage means nothing here when i = 0.

Well, coverage can still tell you that this line is not covered at all. The absence of coverage is a signal, but the presence of it is not sufficient to prove that the code works for all inputs.

We’ve also known this since forever, this has always been Djikstra’s message to software developers.

Re: Coverage is not strongly correlated with test suite effectiveness

#26

Post by tharkun__ »

gravypod wrote: post #21
tharkun__ wrote: post #19

Earlier quoted context omitted.

100% code coverage is still not a good use of everyone's time in most projects and languages. There's a bunch of trivial code that even feels wrong to test. Spend more time cooking up edge case tests that execute some of the branches way more than once. Think of your code like a heat map. Higher heat on lines that get exercised more often by your tests. It's fine that _some_ code has no color at all, while you some o…

> 100% code coverage is still not a good use of everyone's time in most projects and languages The key here is most . Think about your use case before applying anything you read online. > It's fine that _some_ code has no color at all, while you some other paths to be bright red in the end, instead of always just going for a uniform orange for everything. This was the logic tree of a device used for health care work.…

I did write most on purpose, instead of all.

Looks like your case is one where 100% coverage is a good thing and the cost paid for it is absolutely acceptable - nay needed to be paid. And you didn't just go for 100% and stop there because you hit some metric but you actually did the thing that's more important too ("data coverage"). Kudos!

Re: Coverage is not strongly correlated with test suite effectiveness

#27

Post by _6pvr »

tharkun__ wrote: post #10

In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other peop…

> Is it just me or was this _not_ surprising at all?

It wasn't surprising to anyone that has reflected about the value of tests. Mindless testing/TDD isn't usually reflective, though.

Re: Coverage is not strongly correlated with test suite effectiveness

#28

Post by icedchai »

I was on a team that needed to write tests to deploy their app on an "enterprise CRM" that you probably all have heard of. You couldn't upload the app for distribution unless you had X% test coverage. Of course, nobody wants to write all these tests.

We had a few real tests. But most of our tests would call methods and check for != null. This worked great, got us up past 70% coverage with little effort. Supposedly these apps went through a "review" of some sort before they were released to the public. Oddly, our test methodology was never an issue.

Re: Coverage is not strongly correlated with test suite effectiveness

#29

Post by KZeillmann »

A team getting to 100% test coverage and enforcing it feels like an application of Goodhart's Law.

When a measure becomes a target, it ceases to be a good measure.

At my last job, we required 100% code coverage for most code, and the other parts of the code (in ideality) were marked with ignores that were well-thought-through.

In practice, I ended up writing a bunch of test cases only to hit the if blocks. It didn't mean that the tests I wrote were any good. I'm still a fan of 100% test coverage, but with some leeway for what needs to be ignored, and what needs to be tested later (non-critical paths, if you have a deadline, perhaps)

Re: Coverage is not strongly correlated with test suite effectiveness

#30

Post by spmurrayzzz »

tharkun__ wrote: post #10

In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other peop…

Like most others here in the comments, I find the results not particularly controversial. Pretty easy to imagine scenarios where lines of code are run, but the tests themselves are suboptimal or incomplete.

As an aside— I've often mused at to whether theres a more useful multivariate (but still coarse) measure that could combine coverage with cyclomatic complexity and number of tests/assertions. Seems like it would be incrementally better than just coverage alone.

Post reply on HN