It's a case of convention over common sense.
The tragedy of 100% code coverage (2016)
91–100 of 346 posts
Re: The tragedy of 100% code coverage (2016)
#92I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…
Tests are not meant for ensuring it works with all inputs. You cant simply just throw values at it hoping its all okay. To prove it works with all possible inputs, there are other tools at your disposal.
What tools would that be?
I'd like to hear about real world test scenarios where all possible inputs are tested.
Re: The tragedy of 100% code coverage (2016)
#93I use the following list to decide on creating a unit test or not. More yeses means a unit test is a good idea.
1. Is it hard to instantly test the code when implementing it? (Might be the case for library code)
2. Is there a chance the underlying implementation might change (and so might break in the future)?
3. Will the interface of the class remain stable? (If not unit test needs to be rewritten too)
4. Will functional tests pass when something breaks in this class?
Re: The tragedy of 100% code coverage (2016)
#94My main issue with unit testing is what defines a unit? Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression. Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box. That…
I agree with you. That's called functional testing, and it is very useful, but it is not unit testing. Unit testing: Test all methods and paths of a class, even private ones. Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests. ROI of functional testing is high, as it is usually done with real data. In m…
I think that depends on the code base, I agree with your points for more procedural code bases, but for functional or OO code you should be able to test most of your logic without mocks or wiring too much up.
Re: The tragedy of 100% code coverage (2016)
#95I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…
I don't see how that's a "tragedy", as in "an event causing great suffering, destruction, and distress, such as a serious accident, crime, or natural catastrophe".
You're also making it sound like somebody promised you that tests can prove the absence of bugs, when that was never the bargain and smart people have already told you so, probably before you were born even.
> Testing shows the presence, not the absence of bugs
Dijkstra (1969)
Re: The tragedy of 100% code coverage (2016)
#96I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…
I agree that test coverage does not ensure that the code is really tested. But not covered code is not tested code.
So i use coverage in this way, to spot not tested code.
Re: The tragedy of 100% code coverage (2016)
#97I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…
I think that mutation testing should replace code coverage. It really ensure that the test verify the code behavior and not only invoked it.
Re: The tragedy of 100% code coverage (2016)
#98Earlier quoted context omitted.
It's strange to me that we continue to make languages that force us to make certain architectural choices to help facilitate testing.
Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)
And it is built in in D:
http://dlang.org/spec/unittest.html
It's been very effective at improving the overall quality of the code. And because of CTFE (Compile Time Function Execution) and static assert, many code correctness tests can even be run at compile time.
Re: The tragedy of 100% code coverage (2016)
#99Earlier quoted context omitted.
I agree with you. That's called functional testing, and it is very useful, but it is not unit testing. Unit testing: Test all methods and paths of a class, even private ones. Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests. ROI of functional testing is high, as it is usually done with real data. In m…
Some counterpoints: - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests. - If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful. - Yes, integration tes…
> - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests.
I think we talk about the same thing, sometimes I will test something internal.
I've thought about what I'm doing as considering larger "units" in my unit testing, but perhaps "functional testing" as parent introduced is what I'm advocating. I see this as distinct from integration testing. In a micro service architecture, my integration test would be chaining multiple services together into test scenarios.
Re: The tragedy of 100% code coverage (2016)
#100I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…
I think that mutation testing should replace code coverage. It really ensure that the test verify the code behavior and not only invoked it.
The trouble with mutation testing is that most mutations will completely break an application.
Why not prove the code is correct instead? Should be much cheaper than 100% coverage and more certain.