Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

141–150 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#141

Post by biztos »

A lot of people here seem to have strong opinions against 100% coverage, so I'll risk their ire with my strong opinion in favor.

If you have, say, 95% coverage -- and most corporate dev orgs would be thrilled with that number -- and then you commit some new code (with tests) and are still at 95%, you don't know anything about your new code's coverage until you dig into the coverage report. Because your changes could have had 100% coverage of your new thing but masked a path that was previously tested; or had 10% but exercised some of the previously missing 5%.

If you have 100% coverage and you stay at 100% then you know the coverage of your new code: it's 100%. Among other things this lets you use a fall in coverage as a trigger: to block a merge, to go read a coverage report, whatever you think it warrants.

Also, as has been noted elsewhere, anything other than a 100% goal means somebody decides what's "worth" testing... and then you have either unpredictable behavior (what's obvious to whom?) or a set of policies about it, which can quickly become more onerous than a goal of 100%.

It's important to remember that the 100% goal isn't going to save you from bad tests or bad code. It's possible to cheat on the testing as well, and tests need code review too. There's no magic bullet, you still need people who care about their work.

I realize this might not work everywhere, but what I shoot for is 100% coverage using only the public API, with heavy use of mock classes and objects for anything not directly under test and/or not stable in real life. If we can't exercise the code through the public API's then it usually turns out we either didn't rig up the tests right, or the code itself is poorly designed. Fixing either or both is always a good thing.

I don't always hit the 100% goal, especially with legacy code. But it remains the goal, and I haven't seen any convincing arguments against it yet.

Open the flame bay doors, Hal... :-)

Re: The tragedy of 100% code coverage (2016)

#142

Post by saltedmd5 »

dahauns wrote: post #137
saltedmd5 wrote: post #129

The big error being made in this article (and most of the comments here) is the assumption that the purpose of unit tests is to "catch bugs." It isn't. The purpose of unit tests is to document the intended behaviour of a unit/component (which is not necessarily a single function/method in isolation) in such a way that if someone comes along and makes a change that alters specified behaviour, they are aware that they…

The big error being made in this article (and most of the comments here) is the assumption that the purpose of unit tests is to "catch bugs." More like "in countless articles, comments, talks and projects over, say, the last decade".

Yes - it's pretty frustrating to try and have an important discussion when 99% of the time it is framed incorrectly from the start.

Re: The tragedy of 100% code coverage (2016)

#143

Post by yoav_hollander »

One point already made by several people on this thread is that code coverage, while helpful, is not enough (and perhaps is not even the best bang for the buck).

In hardware verification (where I come from, and where the cost of bugs is usually higher), "functional coverage" is considered more important. This is usually achieved via constraint-based randomization (somewhat similar in spirit to QuickCheck, already mentioned in this thread).

I tried to cover (ahem) this whole how-to-use-and-improve-coverage topic in the following post: https://blog.foretellix.com/2016/12/23/verification-coverage...

Re: The tragedy of 100% code coverage (2016)

#144

Post by StavrosK »

bad_user wrote: post #95
cbanek wrote: post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

I don't see how that's a "tragedy", as in " an event causing great suffering, destruction, and distress, such as a serious accident, crime, or natural catastrophe ". You're also making it sound like somebody promised you that tests can prove the absence of bugs, when that was never the bargain and smart people have already told you so, probably before you were born even. > Testing shows the presence, not the absence…

If we're being pedantic, a tragedy is _a drama or literary work in which the main character is brought to ruin or suffers extreme sorrow, especially as a consequence of a tragic flaw, moral weakness, or inability to cope with unfavorable circumstances_.

Definitely not a tragedy.

Re: The tragedy of 100% code coverage (2016)

#145

Post by StavrosK »

Gibbon1 wrote: post #135
kasparsklavins wrote: post #88

Earlier quoted context omitted.

Tests are not meant for ensuring it works with all inputs. You cant simply just throw values at it hoping its all okay. To prove it works with all possible inputs, there are other tools at your disposal.

I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible. Brute force. Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results. Me I have a pi…

How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.

Re: The tragedy of 100% code coverage (2016)

#146

Post by alkonaut »

camus2 wrote: post #136
iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

> The worse the developer, the more tests he'll write. Spare us your platitudes. Most developers don't decide the code coverage, the manager or the client does.

The manager is hopefully also a developer, and listens to his team Else you have a manager that sets arbitrary quality metrics that the dev team doesn't agree are useful then - and that's a way bigger problem.

In the case of contractor work where a customer actually buys the code and not just the functinoality - it's very tricky. I have never done contractor work so I'm not aware of how contracts are usually written. How do you set a quality metric? I'd be much more comfortable to agree to third party to judge the quality than to have an arbitrary metric in a contract for e.g. ratio number of comments to code lines, the average number of character in symbol names or the percentage of lines of code covered by tests.

Re: The tragedy of 100% code coverage (2016)

#147

Post by FroshKiller »

dublinclontarf wrote: post #124
iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

I inherited a code base with LOTS of test coverage. Made one small change and everything broke. Looked into the tests, oh the humanity!!! Dropped the test suite entirely, haven't had any issues since. We have VERY good logging and notifications though so anything goes wrong we know all about it.

You know all about it at run-time.

Re: The tragedy of 100% code coverage (2016)

#148

Post by crdoconnor »

phlakaton wrote: post #61
ardit33 wrote: post #45

Earlier quoted context omitted.

I agree with you. That's called functional testing, and it is very useful, but it is not unit testing. Unit testing: Test all methods and paths of a class, even private ones. Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests. ROI of functional testing is high, as it is usually done with real data. In m…

Some counterpoints: - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests. - If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful. - Yes, integration tes…

"If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests."

This is actually where doctests work pretty well as they both document and test.

"If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful."

I don't find tracking down bugs with a repeatable test case to be much of a problem (events in live systems are a different story). It becomes even less of a problem if you sprinkle some assertions around the code that block off invalid code paths.

"Yes, integration tests can be mocked up to look like real world data, and of course you can even feed them real data. The flip side is that their data requirements can be heavy, and they can be quite cumbersome to set up."

Building mocks in unit tests is usually even more cumbersome and tedious.

For overly large sets of real world data I've had some success taking live database snapshots and cutting out 95% of the data before using the cut down to size dump for testing.

Re: The tragedy of 100% code coverage (2016)

#149

Post by alkonaut »

koonsolo wrote: post #93

I use the following list to decide on creating a unit test or not. More yeses means a unit test is a good idea. 1. Is it hard to instantly test the code when implementing it? (Might be the case for library code) 2. Is there a chance the underlying implementation might change (and so might break in the future)? 3. Will the interface of the class remain stable? (If not unit test needs to be rewritten too) 4. Will funct…

I often try to write a unit test just to see whether it's possible. If not (i.e. one of the questions was no) there is always a chance that the design can be improved. E.g. can the interface be minimized to something that is stable, and simpler to test?

Testable code tends to be better (have less dependencies, less state etc) than non testable code. So making it testable is a goal in itself, even if the tests are never even committed.

Re: The tragedy of 100% code coverage (2016)

#150

Post by FroshKiller »

quantumhobbit wrote: post #6
ioquatix wrote: post #3

I have 100% code coverage on a couple of projects. It has two benefits: Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected. New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the…

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…

A "pointless" test of a getter or setter could save you a lot of trouble later after someone introduces a side effect.
Post reply on HN