Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

Post reply on HN
Print view View original

41–50 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#41

Post by tharkun__ »

_6pvr wrote: post #35
fiddlerwoaroof wrote: post #32

Earlier quoted context omitted.

It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.

> It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement. Tests are code. Code can be sloppy, fallible, useless. Writing a consumer before you write a provider doesn't make either more robust, just the point at which they meet more clear. You can certainly write a test that uses a function, but the test doe…

Very well said.

I have nothing against TDD if that's something that helps a particular person write good code and good tests.

I find though that people that write good tests are just people that write good tests. Most people write tests that assert on implementation details instead of inputs and output and that's easily doable via TDD as well.

Re: Coverage is not strongly correlated with test suite effectiveness

#42

Post by jeffbee »

lwansbrough wrote: post #36

Now, I've never worked at a big company with lots of developers, but testing seems dramatically overvalued. My company currently employs zero testing (meaning zero automated tests). Anything that could negatively impact the company if it blew up is examined pretty closely and then set loose. Occasionally things break. Our company serves tens of millions of users monthly. We have bugs, we fix them as needed. We're not…

I am a huge fan of unit testing and the best thing about TDD is getting your function signatures and other APIs sorted out before writing your implementation. Having to write the calling function first helps ensure that the called function has a signature that will be useful to other callers, instead of something awkward that seemed OK when you started writing the implementation but later turned out to be imperfect.

Also, TDD wards off the problem of people who write untestable code. A lot of functions simply cannot be tested. TDD eliminates that problem.

Re: Coverage is not strongly correlated with test suite effectiveness

#43

Post by thaumasiotes »

jacobsenscott wrote: post #20

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…

> But once it is "covered" the coverage report is useless.

Well, yes, any report that is guaranteed to say the same thing every time is useless.

Re: Coverage is not strongly correlated with test suite effectiveness

#44

Post by fiddlerwoaroof »

merb wrote: post #38
fiddlerwoaroof wrote: post #33

Earlier quoted context omitted.

It’s not a problem here, but if the type doesn’t guarantee reference equality for equal value, you’d also need a test to differentiate between a == b And (in Java) a.equals(b) In order to catch misguided refactorings. (I’ve introduced bugs this way before I learned that == can give a false positive for strings if the strings are interned)

in java the method that takes two `int`s would still use ==. java has value types which don't need `equals`.

Yeah, that’s why I said “it’s not a problem here”.

Re: Coverage is not strongly correlated with test suite effectiveness

#45

Post by thaumasiotes »

tharkun__ wrote: post #10

In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other peop…

> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input?

How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input.

Now, if your code is just pumping data from one point to another point, it's possible that the destination will vary its behavior based on the input in a way that your coverage-based testing can't see. But if you're doing something with the input yourself, then 100% coverage means you tested every possible kind of input.

(Actually, a "branching" problem can still arise if you have something like

  if( condition1 ) {
    do_stuff();
  }
  
  if( condition2 ) {
    do_other_stuff();
  }
  
  if( condition3 ) {
    do_other_other_stuff();
  }
Because with this code, it's possible to get test coverage of every line of code (there are 6 of them, not counting the '}'s) while not covering every branch (there are 8 of them, depending on the combined truth values of condition1, condition2, and condition3).)

Re: Coverage is not strongly correlated with test suite effectiveness

#46

Post by [deleted] »

throwawaylinux wrote: post #23
falcolas wrote: post #7

This makes perfect sense: a simple function with two code paths that splits on the comparison of two signed integers immediately requires a minimum of three test cases for correctness, yet it only takes two to achieve 100% code coverage. Checking for correctness for corner case values - maxint, minint, zero - adds a minimum of another 9 cases. And it will take many, many more test cases if you're working with a weakl…

Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are…

[deleted]

Re: Coverage is not strongly correlated with test suite effectiveness

#47

Post by fiddlerwoaroof »

tharkun__ wrote: post #41
_6pvr wrote: post #35

Earlier quoted context omitted.

> It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement. Tests are code. Code can be sloppy, fallible, useless. Writing a consumer before you write a provider doesn't make either more robust, just the point at which they meet more clear. You can certainly write a test that uses a function, but the test doe…

Very well said. I have nothing against TDD if that's something that helps a particular person write good code and good tests. I find though that people that write good tests are just people that write good tests. Most people write tests that assert on implementation details instead of inputs and output and that's easily doable via TDD as well.

It’s hard to assert based on implementation details if you don’t have an implementation yet: the advantage of test-first code is that you usually have to write the tests in terms of the interfaces of the input and output types rather than inspecting the code under test.

Re: Coverage is not strongly correlated with test suite effectiveness

#48

Post by lwansbrough »

jeffbee wrote: post #42
lwansbrough wrote: post #36

Now, I've never worked at a big company with lots of developers, but testing seems dramatically overvalued. My company currently employs zero testing (meaning zero automated tests). Anything that could negatively impact the company if it blew up is examined pretty closely and then set loose. Occasionally things break. Our company serves tens of millions of users monthly. We have bugs, we fix them as needed. We're not…

I am a huge fan of unit testing and the best thing about TDD is getting your function signatures and other APIs sorted out before writing your implementation. Having to write the calling function first helps ensure that the called function has a signature that will be useful to other callers, instead of something awkward that seemed OK when you started writing the implementation but later turned out to be imperfect.…

The notion that you can get software right the first time seems... a bit naive to me, regardless of the methodology, respectfully. We solved the internal API problems in our app simply by either versioning internal services/APIs (ie. MyServiceClassV2), adding a new method signature and deprecating the old one, or updating the existing method signature, which is pretty safe in strictly typed languages. (And if you write code that uses reflection, you get to be on call when it breaks.)

Re: Coverage is not strongly correlated with test suite effectiveness

#49

Post by sverhagen »

taeric wrote: post #37
jacobsenscott wrote: post #20

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…

I was going to be worried if mutation testing wasn't mentioned here. Is a great way to test the effectiveness of your tests at catching the common mistakes people make in code. That is, a mutation suit doesn't test your code, per se. It tests your test suit.

I've worked a lot on Java, and while there is great mutation testing tools (well, PIT specifically), I find them hard to "scale" practically, i.e. to run them in an automated fashion "every time". You either get long running builds with them in it, or you have to deal with the logistics of moving caches around (so that the mutation testing can be incremental). And as just another tool that a developer MAY use if they so choose to, it's... well... nice... but no longer front of mind, for most people on the team. I'm very interested if you're experiences have been different?
Post reply on HN