Even better are claims of data coverage. USER: And every possible combination of data has been tested and validated to be correct like your management assured us? PROGRAMMER: No, that's simply not possible. For example, if there are 10,000 possible values for variable one, 1,000,000 values for variable two, and 5,000 values for variable three, that'd be 50 trillion possible combinations. And you have thousands of variables. USER: OK, but if that's true, why did your management assure us you'd verify every possible combination? PROGRAMMER: Because that's what you wanted to hear?
The Myth of Code Coverage
21–30 of 115 posts
Re: The Myth of Code Coverage
#22Interesting article, but the answer I'd be looking for is #itdepends.
What's the impact of a bug in this code base? How likely is there to be a bug? How quickly can you rectify a bug if it happens?
Most of my Makefiles and setup.py files have 0% test coverage. Libraries that are automatically deployed to a few projects, where I can't easily ship a fix and I don't really fully understand the impact - I'd aim for 100%, but not arbitrarily; I want to make sure we've actually tested the edge cases.
Re: The Myth of Code Coverage
#23Earlier quoted context omitted.
Depends on rare errors. I meanthings like 'My database crashed halfway a transaction, my file system drops from under my application, my back end service gave up. You can't do much here. Dump some info, abrt the half- done work, maybe try again somewhere in the future. And yes, that last part migh deserve a test.
If you can't do much, it shouldn't be much code, so coverage should stay high. I used to be of the "70% is good enough" school, but I found that as I wrote better programs, I also achieved better test coverage. Not because I was writing more tests, but because I was choosing core designs with fewer edge cases, simplifying my error handling with fewer reachable paths, and admitting I might as well just crash in more c…
It adds up surprisingly fast. I live mostly in the java world, and libraries throwing 3 or so required to catch exceptions are common.
I'd agree that 90pct is doable for your own algorithmic code. Glue code for libraries, commonly not very well thought out, requires a very different style. Defensive up to the paranoid, and catching every gremlin at the source. It's not uncommon to have more than half of the code dedocated to exception handling. Dumping said libraries is rarely an option, unfortunately.
I'm not necesarily disagreeing with you, but maybe we live in different code worlds.
Re: The Myth of Code Coverage
#24Earlier quoted context omitted.
If you can't do much, it shouldn't be much code, so coverage should stay high. I used to be of the "70% is good enough" school, but I found that as I wrote better programs, I also achieved better test coverage. Not because I was writing more tests, but because I was choosing core designs with fewer edge cases, simplifying my error handling with fewer reachable paths, and admitting I might as well just crash in more c…
It adds up surprisingly fast. I live mostly in the java world, and libraries throwing 3 or so required to catch exceptions are common. I'd agree that 90pct is doable for your own algorithmic code. Glue code for libraries, commonly not very well thought out, requires a very different style. Defensive up to the paranoid, and catching every gremlin at the source. It's not uncommon to have more than half of the code dedo…
We definitely do, and I see how that might apply - I do write some Java but avoid as much as possible the disaster of checked exceptions. Mostly I am talking about Python, Go, TypeScript, and other more esoteric stuff.
I don't agree that this defensive style is necessary though - I do (like everyone these days) write a ton of glue code, and that's as I described. Java is nearly as bad as JavaScript for language-cart-pulling-program-horse designs, it's just a different set of mistakes. It could be so much better if developers just did less.
Re: The Myth of Code Coverage
#25Because code coverage should measure coverage of REQUIREMENTS, not lines of code.
Right coverage metric would take as input list of all requirements broken into smallest conditions and then check if they all are verified by a unit test.
In particular, this would cause almost all code to be covered because if a line of code was not covered by test it would mean it is not necessary to meet any of the requirements.
Interestingly enough, unit tests would also be then complete documentation of every single requirement and that is how I understand "test driven development". Sadly, I have yet to see it implemented anywhere.
Re: The Myth of Code Coverage
#26> 1/3 of the code every software project is irrelevant, buggy, overly complicated, or simply sucks. It has a reason to be where it is, but chances are, one year down the road, it will become a liability. Being dogmatic about tests and covering every line will only make it more difficult to get rid of it.
If you have actually tested every line of code with your test suite (not the same as "covering" every line), then your code better not be irrelevant, buggy, or simply suck. That is the sort of thing you are supposed to uncover and fix as part of the whitebox test development process. If your covered code is that bad, you either haven't actually tested the code or you have completely missed the point of software verification.
Code coverage can be a very misleading metric. I have to clarify the difference between "tested" and "covered" code to coworkers and management all the time. In the same way that "standard lines of code" is an insufficient measurement of complexity or development effort, "coverage" is an insufficient measurement of test quality. However, it can still help with ballpark estimates.
Uncovered code is actually a much more useful metric, IMO. You can cover code without testing it, but there is no way you could have tested uncovered code.
I think the point the author should be making here is that not every app needs to be 100% tested.
Re: The Myth of Code Coverage
#27I'm a strident adherent to having 100% code coverage, but this mostly works because my history in infrastructure and a desire to sleep well. The key idea that I have found that if you can't synthetically get your process into a specific state, then life is going to be hard. The problem with a lot of people is that they view testing as a burden rather than a criticism of the code they are testing. If you can't test yo…
You must be trolling.
Re: The Myth of Code Coverage
#28Interesting article, but the answer I'd be looking for is #itdepends. What's the impact of a bug in this code base? How likely is there to be a bug? How quickly can you rectify a bug if it happens? Most of my Makefiles and setup.py files have 0% test coverage. Libraries that are automatically deployed to a few projects, where I can't easily ship a fix and I don't really fully understand the impact - I'd aim for 100%,…
Hrm, when I run `./setup.py test` I run well over 90%, usually 100%, of the lines of code in my `setup.py`. Is this not coverage?
Re: The Myth of Code Coverage
#29Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…
> Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code.
Absolutely correct. But at the same time, Strive for full coverage that hits every line AND tests the appropriate scenarios and edge cases.
Re: The Myth of Code Coverage
#30Interesting article, but the answer I'd be looking for is #itdepends. What's the impact of a bug in this code base? How likely is there to be a bug? How quickly can you rectify a bug if it happens? Most of my Makefiles and setup.py files have 0% test coverage. Libraries that are automatically deployed to a few projects, where I can't easily ship a fix and I don't really fully understand the impact - I'd aim for 100%,…
Hrm, when I run `./setup.py test` I run well over 90%, usually 100%, of the lines of code in my `setup.py`. Is this not coverage?
:) python3.9 -m pytest for me. My point was more that not every line of code should be treated the same way.