I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…
I agree that test coverage does not ensure that the code is really tested. But not covered code is not tested code. So i use coverage in this way, to spot not tested code.
The tragedy of 100% code coverage (2016)
221–230 of 346 posts
Re: The tragedy of 100% code coverage (2016)
#222Always write tests. And strive for maxmium coverage. But make sure you write the right kinds of tests:
- Don't overuse mocks. Mocks don't represent real conditions you would actually see in production. Favor using real dependencies over mocks.
- Don't overspecify your tests. Test only publicly specified parts of the contract. Things that you need to be true and that the callers of the module expect to be true. And yes, you will change the test when the contract changes.
Re: The tragedy of 100% code coverage (2016)
#223Earlier quoted context omitted.
So you're saying test coverage is a negative? It's not an either or.
If I am understanding correctly, he is saying that well-written code requires fewer tests in order to be fully tested. Fully tested is better than untested, assuming that code is fully tested, well written code will require fewer tests. That is my understanding, as it is rather hard to extract a solid argument from his posts.
Re: The tragedy of 100% code coverage (2016)
#224Earlier quoted context omitted.
So you're saying test coverage is a negative? It's not an either or.
If I am understanding correctly, he is saying that well-written code requires fewer tests in order to be fully tested. Fully tested is better than untested, assuming that code is fully tested, well written code will require fewer tests. That is my understanding, as it is rather hard to extract a solid argument from his posts.
Re: The tragedy of 100% code coverage (2016)
#225Laziness is a kind of populism. Such articles will be always upvoted.
Working smarter isn't always in opposition to working harder.
Re: The tragedy of 100% code coverage (2016)
#226He's right, but he's conflating 100% code coverage with using mocks with writing tests. Always write tests. And strive for maxmium coverage. But make sure you write the right kinds of tests: - Don't overuse mocks. Mocks don't represent real conditions you would actually see in production. Favor using real dependencies over mocks. - Don't overspecify your tests. Test only publicly specified parts of the contract. Thin…
Re: The tragedy of 100% code coverage (2016)
#227Earlier quoted context omitted.
> The worse the developer, the more tests he'll write. This is not only wrong, but also dangerous. I agree with the second part of your comment (about being pedantic about the number of unit tests), but a good and comprehensive test suite is fundamental for large projects, especially projects with many moving parts and a large turn-over of people (basically any big enterprise).
By far the best codebase I ever worked with had zero test cases, was nearly 30 years olds, been worked on by not just several teams but several companies, had basically zero documentation, and was still easy to read and understand. Unit tests only really catch a tiny fraction of bugs relating to defective code, not poor design, poor market fit, poor understanding of requirements, or any of the other bugs that most of…
That's because unit tests aren't supposed to be a magical silver bullet that automatically fixes everything. Unit tests are like an exercise regimen. Just because you work your ass off in the gym doesn't mean that you'll get the result you want. If you're still eating a massive amount of calories, have inconsistent workout patterns, or do exercises incorrectly, you will only see minimal gains. Likewise, if your unit tests are not testing the correct logic, it's just an exercise in futility.
People make jokes about the infamous null test:
Person p = new Person();
p.setName("a");
p.setAge(0);
assertNotNull(p);
assertNotNull(p.getName());
assertNotNull(p.getAge());
Yet there are a non-zero number of production code bases with those kinds of tests attached to them. And I firmly believe that this is one of the drivers of hatred towards unit tests: people see poor unit tests like the above, so they instinctively write off unit tests as a waste of time.Just because someone uses a tool incorrectly doesn't mean that the tool does not have value. A spudger isn't going to drive a nail into the wall, but that's not what a spudger was designed for.
Re: The tragedy of 100% code coverage (2016)
#228I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…
I only have one piece of code for which I can say true 100% coverage exists: a library that works with HTML/CSS color values, and which ships a test that generates all 16,777,216 hexadecimal and integer rgb() color values, and runs some functions with each value. However, I don't run that as part of the normal test suite. It only gets run when I'm prepping a new release, as a final verification step; the normal runs-…
Re: The tragedy of 100% code coverage (2016)
#229The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…
no.... please..... no..... let's not go down this path. Why does everything have to be extreme? No tests, 100% tests... it's all bollocks.
The most robust code I've ever written was 100% because of unit tests. It was a little engine that approved the prior authorization for medications. The unit tests didn't cover 100% of the application. In fact the only bit it did cover was the various use cases in the approval logic. The tests were invaluable in writing the logic, and continued to be invaluable for the maintenance of the project.
Tests are a tool. You don't need to use the tool for EVERYTHING, but sometimes it's the best tool for the job. Using it doesn't make you a bad developer, using it badly makes you a bad developer.
Re: The tragedy of 100% code coverage (2016)
#230Not sure if this is already mentioned but for me the most concise illustration of this fallacy was in The Pragmatic Programmer book. They had a function like this: double f( double x ) { return 1/ x; } They pointed out that it is trivial to get 100% coverage in test cases but unless your tests include passing in 0 as the parameter you are going to miss an error case.
passing 0 to f returns infinity (IEEE 754) Not arguing the point here. It's just a terrible example.