Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

101–110 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#101

I find it somewhat rare that I update code and replace an implementation in a way that tests immediately pass. That said, I think the value of unit tests isn't so much in the coverage metric as much as 1) showing someone did the legwork to test the code 2) document some amount of caveats and expected behavior, and 3) provide the next person to edit the implementation a small test "framework" (mocks, dummy data) to bu…

For some codebases I think tests breaking at every change is the expected behavior.

In a way it gives an approximation of the dependency graph of the code worked on, and acts as a reminder of the other functionalities that one might not have intended to touch.

Re: Coverage is not strongly correlated with test suite effectiveness

#102
I've always wondered about building a test-sensitivity metric to complement coverage (and thus make coverage more useful):

- Pick a random line of code (skip comments and whitespace if you can)

- Delete/modify it in some way

- Run the test suite and see if it passes

Do this thousands of times on a codebase, then report on the test suite failure rate accross all of the random changes, broken down by file etc.

You could track it all in a long running database and focus each test on lines which have recently changed to make it less expensive.

I've never actually done this, but I'd guess that you could get a good sense of which files have good tests, and (combined with traditional coverage data) which files have bad tests vs just no tests.

Re: Coverage is not strongly correlated with test suite effectiveness

#103
post #87
post #69

Aiming for 100% test coverage actually produces negative value. You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage for a few months and you will see it for yourself. The negative value comes from: 1. The time wasted on writing all these tests that are mostly ceremonious in nature. But, more importantly: 2. It makes refactoring a big pain in the ass. Why? Because 100% test c…

3. Your code base may start to become contorted. I've seen good programmers create bogus classes to allow test-time mocking, or add oddball env vars and configurations to let the test harness manually reach every last line. Even if that line is not worth testing: if(!(x=malloc(BUF_SIZ)) || ENV[TEST_MEM_FAIL_12]) { exit(1); } Tying code and tests this tightly discourages refactoring. Another example: a different code…

"Mocking" is such a weird thing to me and I don't think it serves a good purpose. It's the kind of thing that would only arise if you assume a-priori that 100% test coverage is a non-negotiable must.

If you have a function A that calls B to get some data (by doing I/O) then process it using C, then the 100% cov rule would force you to mock B when you test A. But then what is the value of this test? What guarantees is it giving you? You are basically testing the internals of A, you are testing the implementation details. You are testing that calling A is the same as calling B and passing the result to C.

Re: Coverage is not strongly correlated with test suite effectiveness

#104

I've always wondered about building a test-sensitivity metric to complement coverage (and thus make coverage more useful): - Pick a random line of code (skip comments and whitespace if you can) - Delete/modify it in some way - Run the test suite and see if it passes Do this thousands of times on a codebase, then report on the test suite failure rate accross all of the random changes, broken down by file etc. You coul…

This is known as mutation testing[1].

[1] https://en.m.wikipedia.org/wiki/Mutation_testing

Re: Coverage is not strongly correlated with test suite effectiveness

#105

I've always wondered about building a test-sensitivity metric to complement coverage (and thus make coverage more useful): - Pick a random line of code (skip comments and whitespace if you can) - Delete/modify it in some way - Run the test suite and see if it passes Do this thousands of times on a codebase, then report on the test suite failure rate accross all of the random changes, broken down by file etc. You coul…

Some especially corectness-concerned or compatibility-concerned people use mutation testing, which changes each line of code and fails the test run if no tests fail as a result of a change.

Re: Coverage is not strongly correlated with test suite effectiveness

#106
Coverage is:

- a negative signal (“low” coverage = no coverage)

- not a positive signal (“high” coverage only means code paths were reached)

- doesn’t mean your tests are thorough

- doesn’t mean your tests are correct

- most importantly for my current work, doesn’t mean you even have an accurate number (istanbul reports 100% coverage if it doesn’t instrument anything; nested languages aren’t instrumented at all, good luck finding out coverage of an XSL transform in JS)

- (still relevant to my work) a good way of conveying safety of small, methodical refactors that can be eyeballed along with incremental changes to see they’re logically equivalent

Re: Coverage is not strongly correlated with test suite effectiveness

#107

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…

> Writing quality tests for existing code is much harder than writing tests for code that doesn't exist

I’ve been trying this and the biggest place it’s helped me is parsing. Working out all the bad formats, types, unexpected stuff before I write the code was worth it.

It’s still probably low quality compared to what it will eventually be, but a pretty good way to start.

Re: Coverage is not strongly correlated with test suite effectiveness

#108

In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other peop…

> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input? How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input. Now, if your code is just pumping data from one point to anoth…

In my experience, when a function has more than 1 if statement, it's probably hard to test and you might want to split it up.

When a function has has more than 2 if statements, you definitely want to break it up.

Imagine the function where conditions aren't related at all, as you posited:

    myfn(int x, int y, string foo) {
        if(foo == "bar") {
            do_stuff();
        }
  
        if( x 
What's really happening here? Why is all this wrapped in 1 function, when the args aren't related at all, nor the work they are dependent on?

Let's say the caller was doing: myfn(1,2,foo);

I would split this function into 3 different function calls in the caller...

    possiblyDoStuff(foo);
    possiblyDoOtherStuff(1,2);
    possiblyDoOtherOtherStuff(time());
Let's use something that you wouldn't simply refactor upstream:

    myfn(string go, string for, string foo) {
        if(go == foo) {
            do_stuff();
        }
  
        if(for == go) {
            do_other_stuff();
        }
  
        if(foo == for) {
            do_other_other_stuff();
        }

        someOtherFn(go, for, foo);
    }
Move that complexity into smaller chunks:

    myfn(string go, string for, string foo) {
        possiblyDoStuff(go, foo);
        possiblyDoOtherStuff(go, for);
        possiblyDoOtherOtherStuff(for, foo);
        someOtherFn(go, for, foo);
    }
Now you have the 7 tests. 2 for each possibly = 6 which are pretty easy and 1 for myfn. It is exceedingly rare that functions require this kind of attention. There are usually a bunch of side effects or return values that are dependent on these checks. eg:

    myfn(string go, string for, string foo) {
        // examples of why you are calling these, to get vals
        var go2 = possiblyDoStuff(go, foo);
        var for2 = possiblyDoOtherStuff(go, for);
        var foo2 = possiblyDoOtherOtherStuff(for, foo);
        return { go2, for2, foo2 };

        //move this to caller someOtherFn(myfn(go, for, foo));
    }

Re: Coverage is not strongly correlated with test suite effectiveness

#109
This is yet another paper where the title exaggerates the importance of the conclusion.

I found a copy of the original paper - there are several risks to the model that the authors mentioned:

- They only mutated conditionals to test coverage, not any of the other possible errors the test suite may be looking for.

- Equivalent mutations may be miscounted, particularly when developers test a lot of off by one errors.

- There data may not meet the assumptions of the Kendell r correlation used.

- Most of their data had low levels of coverage - previous research shows high coverage is needed before it is related to effectiveness.

- They did not account for object oriented code boilerplate code (getter/setters) which do not need to be tested, causing their counts to be potentially be off. This is major, as they were using only Java projects.

- They had very narrow inclusion criteria, so the results may not be generalizable to all codebases. The projects had to have over 1000 test written; the average LOC was generally in the 100k range.

This honestly sounds like some grad students final project, not advice for real world digestion.

Re: Coverage is not strongly correlated with test suite effectiveness

#110

Earlier quoted context omitted.

This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.

Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous. It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments. Contra…

The one thing is a spec doesn't mean the product is correct - you still need testing, just in a different way. It'll probably replace unit tests though.
Post reply on HN