Given unlimited time, money and importance of the code in question - go for 100% coverage. Otherwise, probably not.
Test coverage only matters if it's at 100%
41–50 of 53 posts
Re: Test coverage only matters if it's at 100%
#42> With this setup, test coverage will be at 100% even though we did not test the case where the toaster is not plugged because it is too simple. Writing a test for this "functionality" would not be worth it. Therefore we explicitly exclude it from coverage. You should revisit this decision if it becomes more complex and needs to be tested.
This is ridiculous. He writes unreachable code and says he doesn't test it because it's "too simple". No, you're not testing it because you can't test it, because it's code that will NEVER run. Instead of deleting the pointless code he marks it as not needing test coverage.
Re: Test coverage only matters if it's at 100%
#43100% of what? Oh... line coverage, in Python. Well, with Python, the compiler doesn't do type checking, so 100% coverage (or close) is very important. One of his cases of "what do you not need to check" is error-handling. Yet, having something in production that should raise an exception that gets handled versus having the exception blow up and crash the system is a critical system stability defect. And in fact, in m…
> Well, with Python, the compiler doesn't do type checking, so 100% coverage (or close) is very important.
Intuitively I agree with you: In my personal experience, the more dynamically typed a language is, the more unpleasant surprises I get at runtime.
Some years ago I did a project in MLTON [0], and I was delighted by how many bugs were caught at compile that that, in other languages, would likely go unnoticed until runtime.
Re: Test coverage only matters if it's at 100%
#44As someone who worked somewhere where 100% code coverage was practically required, while it does point out flaws in your testing where you may have "missed a spot." It still doesn't prove that you're testing everything. You never know if a library function you are calling beneath your code has a different path it can also follow. > I believe the only level of test coverage that's worth it is 100% of the lines you wan…
I would go farther and claim that 100% test coverage is worse than 90% test coverage. You're probably testing irrelevant stuff - which just means that you have more test code to maintain, and also likely means that your tests are bad (i.e. not testing business logic, but testing irrelevant implementation details). Last, but not least, 100% test coverage means that your code is bad/ you're not programming defensively…
Yep. At a certain point you start hitting diminishing returns and increased costs.
Re: Test coverage only matters if it's at 100%
#45Not only that, you should subject your automated tests to mutation testing. No link coz wikipedia seems to be down for me.
Re: Test coverage only matters if it's at 100%
#46Earlier quoted context omitted.
And implying that there is 100% coverage just because the instruction register had all valid values is silly. When writing test one should worry about the other registers too ...
Thank you for stating this so succinctly! This concern has been nagging me in the back of my mind, but I hadn't convinced myself that code-coverage was insufficient (in practice) until you framed it this way. Now all kinds of supporting examples come to mind: - divide-by-0 errors - floating-point exceptions - numeric underflow / overflow (both silent and signaled) The alternative code paths exercised by these situati…
Mutation testing can expose weaknesses like these in your automated tests.
Re: Test coverage only matters if it's at 100%
#47Test coverage is a very dangerous metric. IMO it will only be useful the day that we find a way to not take into account those tests that are testing implementation details. Every time that I see a high number being enforced, the end result is always a disaster: lots of poor quality tests that are only there in order to bump up that metric. Those tests are poison: they don't aim to test correctness, they make refacto…
Bingo, this *1000. Your team ends up spending half their time writing painful depressing tests that are bad afterthoughts. Generally with TONS of mocks and little real meaning.
The other half of the time is spent editing the tightly coupled mocks/tests when you refactor the real code. Bad tests can be a liability, not an asset.
Re: Test coverage only matters if it's at 100%
#48As someone who worked somewhere where 100% code coverage was practically required, while it does point out flaws in your testing where you may have "missed a spot." It still doesn't prove that you're testing everything. You never know if a library function you are calling beneath your code has a different path it can also follow. > I believe the only level of test coverage that's worth it is 100% of the lines you wan…
I would go farther and claim that 100% test coverage is worse than 90% test coverage. You're probably testing irrelevant stuff - which just means that you have more test code to maintain, and also likely means that your tests are bad (i.e. not testing business logic, but testing irrelevant implementation details). Last, but not least, 100% test coverage means that your code is bad/ you're not programming defensively…
Or it could mean that you've contorted your code to make sure that you can hit every single defensive edge case, and in the process possibly introduced new bugs due to the increased complexity.
Re: Test coverage only matters if it's at 100%
#49The author is either crazy, or else maybe works in a very niche area where this makes sense. I work in games, and some modules are very conducive to automated testing. Eg. complex math functions, it'd be very easy for them to be not quite right and you would never notice without tests, and also they will probably never change. An area that would be stupid to test (with automated tests) for example, would be a UI menu…
And yet, UI menus are one of the things that always break, and break right in front of the user ("I can't click this button. This is greyed out and it shouldn't be.") Menus should definitely be tested, and I think testing them in an automated way is fine. Without a working UI, you have no control. You can even do model based testing. For example, you're testing a word processor. You know that your state is no documen…
Your sort of reiterating my point.
I know UI stuff breaks a lot, but my point is that when it does break, its obvious. Bugs should be easily caught in human acceptance testing.
Also an automated test UI is never going to be the same as a human test, a lot of UI bugs can result from the vagaries of various physical devices touchscreens / gamepads / motion controls etc.
Also anything that tests such high level functionality is going to be very complicated and touch many parts of your code, it will probably be more complicated and less maintainable than the code its testing, which defeats the purpose imo, do you want to write tests for your tests?
UI stuff tends to change radically over development, so you'd be more than doubling your work with all the testcode youd have to write.
and it would still never replicate an actual human running the actual game, you absolutely are still going to be doing human acceptance tests + the closer you try and get automated tests to that the more complex and less maintainable they get.
Re: Test coverage only matters if it's at 100%
#50Seems like the Intelligent Design nuts arguing "what good is half an eye?". Well, it's half as good as a full eye, a lot better than no eye.