Live data from Hacker News

Test coverage only matters if it's at 100%

dein.fr

21–30 of 53 posts

Re: Test coverage only matters if it's at 100%

#21
We had some mild debates at an old company on this matter (I don't think anyone was really ever pro 100% coverage, but we were under pressure to increase coverage in order to provide momentum and to emphasise that automated testing should be routine)

It resulted in a mug being made in the same style as our company mugs but with the words "Fuck Coverage" instead of the company name.

Eyebrows were raised when it was mistaken for a 'real' mug and taken in to a board meeting by a manager however..

Re: Test coverage only matters if it's at 100%

#22

So, don't bother writing any tests unless you can test everything? At a previous shop, we had a simple test that ensured user could log in for a while, until we created more coverage. We didn't test every aspect of the code. This simple sanity check saved us once or twice. Should we have not written this test just because we couldn't test every conceivable piece of code? Don't let perfect be the enemy of good...

Exactly this, plus, the entire concept of 100% coverage is hilariously wrong.

Are you covering 100% of code interactions? How much of your integration logic are you mocking? Are you covering 100% of HTML and interface logic? How much code is not in your code? What about packages and imports? It's incredibly arbitrary for no good reason except to slow development of a product that may depend on fast development for survival in trade to test for bugs that honestly, actually, may never happen. To be clear, testing is good, but you write tests to serve the code and product, not the other way around.

Re: Test coverage only matters if it's at 100%

#23
The author is either crazy, or else maybe works in a very niche area where this makes sense. I work in games, and some modules are very conducive to automated testing.

Eg. complex math functions, it'd be very easy for them to be not quite right and you would never notice without tests, and also they will probably never change.

An area that would be stupid to test (with automated tests) for example, would be a UI menu.

* The test (ie. replicating a user pressing buttons via code) is probably gonna be more complex and less maintainable than the code its testing.

* If there is a bug it will be obvious in human acceptance testing.

* UI menus are gonna change drastically during development.

Anyway, I could see a misguided requirement for 100% code coverage massively hampering a project and potentially killing it.

Re: Test coverage only matters if it's at 100%

#24
This is the type of article I'd expect from a junior engineer.

Test coverage matters even if it is not 100%, especially if it covers the most important use-cases. Cost and time always needs to be considered. 100% coverage may not be worth it. Coverage based on LOC is also an arbitrary metric in some sense, the author himself has examples of that. From a business POV, use-case coverage is more important.

Re: Test coverage only matters if it's at 100%

#26
Test coverage is a very dangerous metric. IMO it will only be useful the day that we find a way to not take into account those tests that are testing implementation details. Every time that I see a high number being enforced, the end result is always a disaster: lots of poor quality tests that are only there in order to bump up that metric. Those tests are poison: they don't aim to test correctness, they make refactors/improvements a PITA, they just lock you down to a concrete implementation and they don't even give you any guarantees that your code works correctly.

Re: Test coverage only matters if it's at 100%

#27
For me the absence of coverage, or low coverage, is a useful piece of information. It's not the whole story but it is a smell, and it should be taken into consideration with the other smells/signals

Conversely perfect or high coverage is not a positive signal per se, but rather just the absence of a negative signal

Re: Test coverage only matters if it's at 100%

#28

The author is either crazy, or else maybe works in a very niche area where this makes sense. I work in games, and some modules are very conducive to automated testing. Eg. complex math functions, it'd be very easy for them to be not quite right and you would never notice without tests, and also they will probably never change. An area that would be stupid to test (with automated tests) for example, would be a UI menu…

And yet, UI menus are one of the things that always break, and break right in front of the user ("I can't click this button. This is greyed out and it shouldn't be.") Menus should definitely be tested, and I think testing them in an automated way is fine. Without a working UI, you have no control.

You can even do model based testing.

For example, you're testing a word processor. You know that your state is no document. Then you click the open file button. Press some other buttons, and knowing what button it is, you should expect some change.

That being said, usually you can "press" the buttons virtually, for example sending window messages or calling the same functions as the buttons wire up to, rather than knowing the pixel layout.

People have been doing these things for years.

Re: Test coverage only matters if it's at 100%

#29

Earlier quoted context omitted.

Seems like 50% coverage is better than 0% coverage and 75% coverage is better than 50% coverage. The author argues about the horrors of going from 80% to 79.9%, but I don't see how that is any different from the suggestion to explicitly exclude code. The excluded code is still in the gray area of code that is included but not covered by tests. What happens when someone starts making breaking changes to excluded code?

Lots of code bases have code coverage in some middle range, lets say it's 82.34%. What does that number tell you? Did the team write 7.64% more code this month that doesn't have test cases? The point you're making is actually irrelevant to the discussion. If you explicitly excluded code, you probably have a reason for that and you take your chances. Ideally, in a perfect world, you'd have no excluded code.

This is why we have tools like coveralls.

Also, this method doesn't show when you remove excluded code or add coverage for it. In a perfect world you should have no excluded code, and in a perfect world you should have 100% coverage.

World isn't perfect though, and a good working product with critical features is better than one lacking important features but with '100%' coverage.

Re: Test coverage only matters if it's at 100%

#30
100% of what? Oh... line coverage, in Python.

Well, with Python, the compiler doesn't do type checking, so 100% coverage (or close) is very important. One of his cases of "what do you not need to check" is error-handling. Yet, having something in production that should raise an exception that gets handled versus having the exception blow up and crash the system is a critical system stability defect. And in fact, in my personal code, since I tend to try to create exception messages that are as specific as possible, I must admit with chagrin that screwing up a ''.join() to create the exception message is one of my most common errors. I at least touch-test my exception code, without exception.

And then... line coverage? Oh my, that is just the beginning. I pretty much always use the hypothesis package so that everything get exercised with a large variety of inputs. Inputs that I didn't have to think up, and avoid my pre-existing biases of what might get thrown at the function.

Line coverage does do much to really ensure that you have reached "interesting conditions". I wish Python had a good tool for that kind of instrumentation. Line coverage is really only a "touch-test" and mainly useful as a check-in gate, not as production testing.

Post reply on HN