Hillel Wayne discussed the paucity of decent research in software engineering land: https://buttondown.email/hillelwayne/archive/i-ing-hate-scie... (discussion: https://news.ycombinator.com/item?id=27892615)
Coverage is not strongly correlated with test suite effectiveness
31–40 of 178 posts
Re: Coverage is not strongly correlated with test suite effectiveness
#32In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other peop…
> Is it just me or was this _not_ surprising at all? It wasn't surprising to anyone that has reflected about the value of tests. Mindless testing/TDD isn't usually reflective, though.
It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.
Re: Coverage is not strongly correlated with test suite effectiveness
#33This makes perfect sense: a simple function with two code paths that splits on the comparison of two signed integers immediately requires a minimum of three test cases for correctness, yet it only takes two to achieve 100% code coverage. Checking for correctness for corner case values - maxint, minint, zero - adds a minimum of another 9 cases. And it will take many, many more test cases if you're working with a weakl…
Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are…
It’s not a problem here, but if the type doesn’t guarantee reference equality for equal value, you’d also need a test to differentiate between
a == b
a.equals(b)
Re: Coverage is not strongly correlated with test suite effectiveness
#34A team getting to 100% test coverage and enforcing it feels like an application of Goodhart's Law. When a measure becomes a target, it ceases to be a good measure. At my last job, we required 100% code coverage for most code, and the other parts of the code (in ideality) were marked with ignores that were well-thought-through. In practice, I ended up writing a bunch of test cases only to hit the if blocks. It didn't…
It gets crazy with demands for 100% test coverage, in languages where you're encouraged to write compile-time logic, macros, setup-dependent minor details, time bombs, asserts which throw compiler errors when you misconfigure, etc.
Re: Coverage is not strongly correlated with test suite effectiveness
#35Earlier quoted context omitted.
> Is it just me or was this _not_ surprising at all? It wasn't surprising to anyone that has reflected about the value of tests. Mindless testing/TDD isn't usually reflective, though.
It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.
> It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.
Tests are code. Code can be sloppy, fallible, useless. Writing a consumer before you write a provider doesn't make either more robust, just the point at which they meet more clear. You can certainly write a test that uses a function, but the test doesn't actually test anything at all, just that the function exists. aka "mindless TDD".
Re: Coverage is not strongly correlated with test suite effectiveness
#36Now, I've never worked at a big company with lots of developers, but testing seems dramatically overvalued. My company currently employs zero testing (meaning zero automated tests). Anything that could negatively impact the company if it blew up is examined pretty closely and then set loose. Occasionally things break.
Our company serves tens of millions of users monthly. We have bugs, we fix them as needed. We're not writing banking software so the risk profile is pretty small, but I think more companies fall into this category than test engineers would have you believe.
I'd give myself 2 more years in the industry if I had to do TDD, or anything else that strives for even moderate coverage.
Re: Coverage is not strongly correlated with test suite effectiveness
#37It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…
I was going to be worried if mutation testing wasn't mentioned here. Is a great way to test the effectiveness of your tests at catching the common mistakes people make in code.
That is, a mutation suit doesn't test your code, per se. It tests your test suit.
Re: Coverage is not strongly correlated with test suite effectiveness
#38Earlier quoted context omitted.
Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are…
It’s not a problem here, but if the type doesn’t guarantee reference equality for equal value, you’d also need a test to differentiate between a == b And (in Java) a.equals(b) In order to catch misguided refactorings. (I’ve introduced bugs this way before I learned that == can give a false positive for strings if the strings are interned)
in java the method that takes two `int`s would still use ==.
java has value types which don't need `equals`.
Re: Coverage is not strongly correlated with test suite effectiveness
#39This makes perfect sense: a simple function with two code paths that splits on the comparison of two signed integers immediately requires a minimum of three test cases for correctness, yet it only takes two to achieve 100% code coverage. Checking for correctness for corner case values - maxint, minint, zero - adds a minimum of another 9 cases. And it will take many, many more test cases if you're working with a weakl…
Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are…
Let's say I mistype 'a == b' as 'a Let's add 1 to a before the equality comparison in order to meet a new business requirement. fn(1, 1) still works, but does fn(maxint, maxint)? Or does it suddenly throw an exception (or worse, silently roll over to minint)?
Re: Coverage is not strongly correlated with test suite effectiveness
#40Earlier quoted context omitted.
> Is it just me or was this _not_ surprising at all? It wasn't surprising to anyone that has reflected about the value of tests. Mindless testing/TDD isn't usually reflective, though.
It’s hard to do test-first TDD “mindlessly” because writing a test usually forces you to think in terms of the specification of the behavior you’re about to implement.
[deleted]