The tragedy of 100% code coverage (2016)

Has anyone seen a fuzzer that creates variants based on a test suite with 100% coverage? Hmm... the fuzzer still wouldn't necessarily know how to create the correct invariants. #lazyweb

There's use in looking at the testing for industrial strength code e.g. compilers, DirectX, OpenGL conformance (god help you). With that background, I was confused by TDD to put it politely.

This was one of the fun things about working on Midori, the whole system was available to us, components all communicated through well defined interfaces, you really could (if necessary) mock the interfaces above and below and be sure that your logic was being exercised. The browser team, in particular, pushed hard to be close to 100% coverage. The overall number for the project was (IIRC) in the 85% range.

When I'm writing tests, I'm not so concerned about what the tool tells me about which lines are covered, I like to work through the mental exercise of knowing which basis paths are covered. If a function has too many for me to reason about, that is a problem in itself.

As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. If you have many methods in a class with only one basis path, what purpose does the class serve. These testing concerns may be the code smell that points to deeper problems

> As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem.

One other thing about code coverage, unit testing, and other testing fads is I think they actively affect the architecture, and usually in the overthinking it way.

Instead of having one tight bit of procedural code (which may have some state, or some dependency calls), people split it up into multiple classes, and then test each bit. This allows them to use the class architecture to mock things, but really has just multiplied out the amount of code. And in the end, you're running tests over the golden path probably even less. It's even possible to have 100% of the code covered, and not run the golden path, because you're always mocking out at least one bit.

I find that, as I'm building something from scratch, the vast majority of the errors I make are just things I didn't think of. Tests don't help there because I can't test on input that I don't even imagine happening. So I generally write few tests, because, to be honest, most code is trivial and algorithm-light. Sure, if I have to write a parser or something a bit more fiddly, I'll write a unit test to be sure that it's doing what I expect, but that tends to be the exception, not the rule. I do write my code with an eye toward later testability if it turns out to be necessary, but I find that to be fairly easy, and also a good measure of if I'm doing the write thing: most code that isn't testable is probably code that's difficult to read and maintain, anyway, so if I look at something and think "oof, how would I ever write a test for that?" I'll usually delete it and start over.

When I have something that should be working, I test it in a more functional/integrative manner, and move on.

Later, I'll write unit tests when I need to. If I want to refactor something, or drastically change the implementation of something, I'll write out some tests beforehand to be sure that the pre and post behaviors match.

I've always thought that TDD is just premature optimization. You're optimizing for the idea that you -- or someone -- will later need to make large enough changes to your code that you'd worry about breaking it. In my experience that's fairly rare, and you spend less time overall if you just write the tests as you need them, not up-front. Yes, writing a test when the code is fresh in your mind will be faster than writing it much later, but then you're writing a ton of test code that likely won't be necessary.

An objection I hear to this is that you're not just writing tests for yourself, you're writing tests for the others who will need to help maintain your code, perhaps after you're gone. I'm somewhat sympathetic to this, but I would also say that if someone else needs to modify my code, they damn well better first understand it well enough such that they could write tests before changing it (if they deem it necessary). Anything else is just irresponsible.

(Note that I primarily work in strongly statically typed languages. If I were writing anything of complexity in ruby/python/JS/etc., I don't think I'd feel comfortable without testing a lot of things I'd consider trivial in other languages.)

(Also note that some things are just different: if you're writing a crypto library, then you absolutely need to write tests to verify behaviors, in part because you're building something that must conform to a formal spec, or else it's less than worthless.)