The Myth of Code Coverage
preslav.me
The Myth of Code Coverage
1–10 of 115 posts
Re: The Myth of Code Coverage
#2First he says "ideal project", then "1/3 of the code every software project is irrelevant, buggy, overly complicated, or simply sucks".
Doesn't seem ideal to me.
Re: The Myth of Code Coverage
#3I was also going to say around 60-70 pct, but for a different reason: what's left in my code is mostly checking of assertions, debug logging and handling rare errors, i.e code that's not supposed to run. This is code that will dump some state to the logs, then crash without causing corruption.
It tends to be 'tested' accidentaly when I am making big changes, and I add more of it if I get a crash that's undebuggable from dumps and logs.
Re: The Myth of Code Coverage
#4I generally agree with the crux of this and would like to take it a step further. It's far more important to test edge cases and the lesser used endpoints.
We have major parts of our product where the coverage isn't great, but get tested _every time_ you use the product.
The coverage on our login process is mediocre, but every time a Developer or QA person uses the product, they log in. We'd hear about it. Our landing page post login is largely the same story.
Our rarely used tools several clicks deep into the site _all_ have much better coverage.
We don't have continuous deployment and it's not a goal.
Re: The Myth of Code Coverage
#5I was also going to say around 60-70 pct, but for a different reason: what's left in my code is mostly checking of assertions, debug logging and handling rare errors, i.e code that's not supposed to run. This is code that will dump some state to the logs, then crash without causing corruption. It tends to be 'tested' accidentaly when I am making big changes, and I add more of it if I get a crash that's undebuggable f…
> I was also going to say around 60-70 pct, but for a different reason: what's left in my code is mostly checking of assertions, debug logging and handling rare errors, i.e code that's not supposed to run.
I grant you assertion-checking, but code for handling rare errors is not the code you should skip writing unit tests for. If it runs infrequently then you're far less likely to stumble on a regression during other testing, so it's even more important to unit test.
Re: The Myth of Code Coverage
#6Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code.
Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort involved in mocking/wrapping/abstracting sometimes isn’t worth the effort for some code, like simple calls to common libraries, reading a file, CLI help strings, etc. This stuff generally is better served by end-to-end or integration tests. Or just gets exercised naturally as part of development. Or can crash without much consequence.
My point is coverage is a tool that tells you where to look, not the one and only metric that tells you how reliable your code is.
Re: The Myth of Code Coverage
#7I was also going to say around 60-70 pct, but for a different reason: what's left in my code is mostly checking of assertions, debug logging and handling rare errors, i.e code that's not supposed to run. This is code that will dump some state to the logs, then crash without causing corruption. It tends to be 'tested' accidentaly when I am making big changes, and I add more of it if I get a crash that's undebuggable f…
> I was also going to say around 60-70 pct, but for a different reason: what's left in my code is mostly checking of assertions, debug logging and handling rare errors, i.e code that's not supposed to run. I grant you assertion-checking, but code for handling rare errors is not the code you should skip writing unit tests for. If it runs infrequently then you're far less likely to stumble on a regression during other…
Depends on rare errors. I meanthings like 'My database crashed halfway a transaction, my file system drops from under my application, my back end service gave up.
You can't do much here. Dump some info, abrt the half- done work, maybe try again somewhere in the future. And yes, that last part migh deserve a test.
Re: The Myth of Code Coverage
#8Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…
Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs.
Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it.
I highly recommend investing in mutation and fuzzy testing.
Re: The Myth of Code Coverage
#9I'm a strident adherent to having 100% code coverage, but this mostly works because my history in infrastructure and a desire to sleep well. The key idea that I have found that if you can't synthetically get your process into a specific state, then life is going to be hard.
The problem with a lot of people is that they view testing as a burden rather than a criticism of the code they are testing. If you can't test your code easily, then your code is awful. End of story.
Re: The Myth of Code Coverage
#10"Myth" is a strong word, given the author supports 2/3 coverage goals. I call clickbait.