Hmmm....how to count tests? By the number of methods or the number of assertions?
Coverage is not strongly correlated with test suite effectiveness
91–100 of 178 posts
Re: Coverage is not strongly correlated with test suite effectiveness
#92I find it somewhat rare that I update code and replace an implementation in a way that tests immediately pass. That said, I think the value of unit tests isn't so much in the coverage metric as much as 1) showing someone did the legwork to test the code 2) document some amount of caveats and expected behavior, and 3) provide the next person to edit the implementation a small test "framework" (mocks, dummy data) to build on.
Re: Coverage is not strongly correlated with test suite effectiveness
#93Earlier quoted context omitted.
Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'. I'm not saying more tests can't catch specific bugs you might come up with, I'm asking how you can choose numbers that have fundamental edge cases for this specific requirement without looking at the actual implementation. I don't think you really can. maxint/minint/0/-1/1 are generally commo…
Now you understand - testing is extremely close to useless. You can't perfectly choose the test cases for code without looking at its implementation. And by that point, you are just testing the implementation and working backwards. What you want is a specification of the behavior, and to verify that the implementation satisfies the specification. This will be proven in the general case, and then you don't need to thi…
I don't think testing is close to useless, I think it has a lot of value but trying to prove general correctness with your test cases is a fool's errand. Adding test cases intelligently for tricky cases in the specification and/or implementation, hard-to-hit error conditions, or adding tests for bugs you previously fixed, etc is perfectly reasonable without doggedly chasing 100% coverage or 100% correctness.
And yes I'm aware of formal methods, and agree they seem like the better way to go for proving.
Re: Coverage is not strongly correlated with test suite effectiveness
#94Earlier quoted context omitted.
They go hand in hand. It’s obvious that missing branch coverage means your data cases are not exercising all of the edge cases in the code.
Not all code is reachable. Especially things like top-level try/except clauses.
From the entry point, stub a function to throw an exception. You'll reach it. Unit testing isn't about finding every path. Nor is it about writing unit tests for every possible combination of values in a program. A java function that uses an int does not need to test -2147483648 to 2147483647 as inputs. That's not helpful.
Re: Coverage is not strongly correlated with test suite effectiveness
#95Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little abo…
Damn, I love geeking out over testing.
Re: Coverage is not strongly correlated with test suite effectiveness
#96Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little abo…
Yet it's extremely popular to have huge, inefficient test suites running on continuous integration servers hundreds of times a day. It boggles the mind.
This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.
Re: Coverage is not strongly correlated with test suite effectiveness
#97In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other peop…
100% or you suck. You need 100% and then some for FULL coverage.
I work with people who turn in 80% PRs...these people are bug factories.
Re: Coverage is not strongly correlated with test suite effectiveness
#98Earlier quoted context omitted.
Yet it's extremely popular to have huge, inefficient test suites running on continuous integration servers hundreds of times a day. It boggles the mind.
This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.
Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous.
It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments.
Contrast that with something like F*, where the specification and code are right alongside each other, and the implementation gets proven. They are reporting a 5:1 ratio of specification to implementation code lines, which is much better than only a few years ago. Soon I think we’ll be striking distance to largely get rid of huge test suites which are so commonplace today.
Re: Coverage is not strongly correlated with test suite effectiveness
#99I find it somewhat rare that I update code and replace an implementation in a way that tests immediately pass. That said, I think the value of unit tests isn't so much in the coverage metric as much as 1) showing someone did the legwork to test the code 2) document some amount of caveats and expected behavior, and 3) provide the next person to edit the implementation a small test "framework" (mocks, dummy data) to bu…
This.
Re: Coverage is not strongly correlated with test suite effectiveness
#100I think it's easy to conclude code coverage is not a proxy for useful tests. But the lack of any significant degree of code coverage does tell you something. And encountering zero code coverage is certainly a problem. So, there is a continuum here, and a balance to be found.
I have come to realize there are many reasons we test, and they are often in competition with each other for getting developer's time to work on them. (Yes, like the Selfish Gene, having us write more code is simply the way tests get us to write more tests, which is really the end goal of it all.)
Lately I have been toying with the idea of creating different suites of tests for each of these testing concerns, so I could track coverage across them independently. I'm thinking of a) basic proof-of-execution tests, which some people think don't tell you much, but in certain codebases, simply validating that you can build a string of SQL from a varied bunch of parameters and execute it without getting a parsing error provides a significant amount of value, regardless of the results returned. And, you can write that test knowing nothing of the business logic. Another level of tests might be concerned with b) the structure of the method: testing all branches or combinations of parameters. Another suite c) would focus on functional tests. And then you have d) mutation and e) characterization tests. For different parts of the codebase, you might choose to favour increasing coverage for one of these suites over the others. You also might wish to add a suite of security or performance tests.
You don't want to get too granular, but the reason code coverage as a metric is not super helpful is we don't necessarily know what those tests are doing, so the more we can constrain their focus, the more meaningful the code coverage results become. And as a bonus, some of these types of tests are very quick to write (a, d, and e), so you can get quickly get to some useful degree of coverage even on a code base you don't understand, or that has no coverage at all. The value of this is not to be discounted, as writing tests can be very time-consuming, and seem like an impossible task when faced with a large, legacy codebase.