The tragedy of 100% code coverage (2016)
labs.ig.com
The tragedy of 100% code coverage (2016)
1–10 of 346 posts
Re: The tragedy of 100% code coverage (2016)
#2There's a human tendency to overemphasize things you can quantify. So we try to figure out how to test every code path rather than what we should do: try to figure out which inputs we should test against.
Re: The tragedy of 100% code coverage (2016)
#3I have 100% code coverage on a couple of projects. It has two benefits:
Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected.
New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the standard "If your contribution doesn't have 100% coverage it won't be accepted". I don't think this is a bad default position.
Re: The tragedy of 100% code coverage (2016)
#4My version of this is working on a team with 100% coverage that still saw a steady and heavy influx of bugs. 100% coverage does not mean bug free.
I advocate spending time on identifying/inventing the correct abstractions over coverage.
Re: The tragedy of 100% code coverage (2016)
#5There's a human tendency to overemphasize things you can quantify. So we try to figure out how to test every code path rather than what we should do: try to figure out which inputs we should test against.
[deleted]
Re: The tragedy of 100% code coverage (2016)
#6I have 100% code coverage on a couple of projects. It has two benefits: Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected. New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the…
The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security.
So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually test any known edge cases.
Re: The tragedy of 100% code coverage (2016)
#7I wish more people cared about path coverage as opposed to "line coverage".
Re: The tragedy of 100% code coverage (2016)
#8100% code coverage and even TDD'd doesn't and shouldn't mean 100% unit tested. Glue code and declaration doesn't need a unit test. Some functional tests should provide all the coverage needed to give you confidence to refractor that code in the future.
Edit: while I'm a huge TDD advocate, I'm not a big advocate of measuring code coverage. That should only be necessary if you are trying to get a code base under coverage that wasn't TDD'd. Even then I'd rather add the coverage as I'm touching uncovered code. If it works and I'm not touching it, it doesn't need tests.
Re: The tragedy of 100% code coverage (2016)
#9I might be completely wrong on this one, but it seems to me that a lot of the precepts of TDD and full code coverage have a lot to do with the tools that were used by some of the people that popularized this.
Some of my day involves writing Ruby. I find using Ruby without 100% code coverage to be like handling a loaded gun: I can track many outages to things as silly as a typo in an error handling branch that went untested. A single execution isn't even enough for me: I need a whole lot of testing on most of the code to be comfortable.
When I write Scala at work instead, I test algorithms, but a big percentage of my code is untested, and it all feels fine, because while not every piece of code that compiles works, the kind of bugs that I worry about are far smaller, especially if my code is type heavy, instead of building Map[String,Map[String,Int]] or anything like that. 100% code coverage in Scala rarely feels as valuable as in Ruby.
Also different styles make the value of having tests as a way to try to force good factoring changes by language and paradigm. Most functional Scala doesn't really need redesigning to make it easy to test: Functions without side effects are easy, and are easier to refactor. A deep Ruby inheritance tree with some unnecessary monkey patching just demands testing in comparison, and writing the tests themselves forces better design.
The author's code is Java, and there 95% of the reason for testing that isn't purely based on business requirements comes from runtime dependency injection systems that want you to put mutability everywhere. Those are reasons why 100% code coverage can still sell in a Java shop (I sure worked in some that used too many of the frameworks popular in the 00s), but in practice, there's many cases where the cost of the test is higher than the possible reward.
So if you ask me, whether 100% code coverage is a good idea or not depends a whole lot on your other tooling, and I think we should be moving towards situations where we want to write fewer tests.
Re: The tragedy of 100% code coverage (2016)
#10I have 100% code coverage on a couple of projects. It has two benefits: Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected. New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the…
The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…
Exactly. It's like testing a newly built bridge by crossing it a hundred times with a bike. Sure, you "covered" it, but you sure as hell did not test it.