Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

51–60 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#51

Post by defined »

quantumhobbit wrote: post #6
ioquatix wrote: post #3

I have 100% code coverage on a couple of projects. It has two benefits: Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected. New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the…

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…

> The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security.

This is true.

But isn't that true of any test? A suite of naive or badly written tests can also give us a false sense of security, so why write any at all (I don't mean that literally)?

I think that a greater level of confidence in our code can be achieved by a combination of

- Judicious choices of unit and integration tests

- Static and dynamic analysis (if the language supports it), and

- Property-based testing (the canonical example of which is QuickCheck[1]). Property-based testing is a great way to help us hit those edge cases.

As for 100% code coverage, I think it is worth striving for, not just for the sake of having all lines of code tested. It can expose design and testability flaws, for example.

If 100% coverage cannot be achieved, we need to ask ourselves:

- Did we really need that piece of code we couldn't test? Is it actually called anywhere, or is it one of those YAGNI things?

- Did we write code that is not very testable? Are there functions or methods, for example, that are so dependent on external state that they can't be mocked or tested some other way? Should we refactor it?

- If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it. More severe defects have probably been caused by a single line of untested code[2][3] than we may suspect.

- If it's a getter or setter, and it's not used (and therefore code covered) in other test cases, maybe it's superfluous and should be removed.

References

[1]: https://en.m.wikipedia.org/wiki/QuickCheck

[2]: https://www.imperialviolet.org/2014/02/22/applebug.html

[3]: http://users.csc.calpoly.edu/~jdalbey/SWE/Papers/att_collaps...

Re: The tragedy of 100% code coverage (2016)

#52

Post by pacaro »

vincnetas wrote: post #47
cbanek wrote: post #36

Earlier quoted context omitted.

Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)

What are the languages that have testing not as afterthought?

This depends on how you think of things. Some languages (Eiffel?) have pre- and post-conditions, which do some of this job. Much of the boilerplate testing done for dynamic languages is done by the compiler for static languages. The borrow checker in Rust is doing the work that a test suite and/or static analysis tool would be doing for C/C++

Re: The tragedy of 100% code coverage (2016)

#53

Post by ruleabidinguser »

mikestew wrote: post #28

The tragedy of 100% code coverage is that it's a poor ROI. One of things that stuck with me going on twenty years later is something from an IBM study that said 70% is where the biggest bang-for-the-buck is. Now maybe you might convince me that something like Ruby needs 100% coverage, and I'd agree with you since some typing errors (for example) are only going to come up at runtime. But a compiled (for some definitio…

70% coverage... I don't know if I'm going to be able to give up my 0% coverage 1-man projects...

Re: The tragedy of 100% code coverage (2016)

#54

Post by pacaro »

mdpopescu wrote: post #33
pacaro wrote: post #25

Earlier quoted context omitted.

This was one of the fun things about working on Midori, the whole system was available to us, components all communicated through well defined interfaces, you really could (if necessary) mock the interfaces above and below and be sure that your logic was being exercised. The browser team, in particular, pushed hard to be close to 100% coverage. The overall number for the project was (IIRC) in the 85% range. When I'm…

> If you have many methods in a class with only one basis path, what purpose does the class serve. It ties together related algorithms. You want "addDays(n)", "addHours(n)", "addMinutes(n)" and so on to be in the same class, even if they're one-liners.

Clearly there are good examples either way. I'm more interested in the idea that this strong mismatch between the complexity of the code and the complexity of its tests is, in itself, a code smell that may point to an issue that is nothing to do with TDD.

I note that the examples you give have trivial test cases - presuming you don't care about overflow, and if you do, then those methods now have basis paths, whether explicit or implicit depends on the language.

Re: The tragedy of 100% code coverage (2016)

#56

Post by ruleabidinguser »

devrandomguy wrote: post #26
penpapersw wrote: post #16

I think a bigger epidemic is we're putting too much emphasis on "do this" and "do that" and "if you don't do this then you're a terrible programmer". While that sometimes may be true, much more importantly is to have competent, properly trained professionals, who can reason and think critically about what they're doing, and who have a few years of experience doing this under their belt. Just like other skilled trades…

It seems to me that management is taught that a dependence on expensive experts, is a problem to be optimized. They want to manage the development process in a way that allows them to easily swap out one developer or team for another, or to ramp up production simply by increasing the number of developers assigned to a task. It is almost as if they see the success that we have had in dev/ops with the "cattle, not pets…

Will they succeed?

Re: The tragedy of 100% code coverage (2016)

#57

Post by vacri »

eksemplar wrote: post #46

We've almost stopped unit testing. We still test functionality automatically before releasing anything into production, but we're not doing a unit test in most cases Our productivity is way up and our failure rates haven't changed. It's increased our time spent debugging, but not by as much as we had estimated that it would. I won't pretend that's a good decision for everyone. But I do think people take test-driven-d…

At one place I work, the target is 80% rather than 100%. Seems to be a saner target than full coverage.

Re: The tragedy of 100% code coverage (2016)

#58

Post by saosebastiao »

thinkloop wrote: post #49
cbanek wrote: post #36

Earlier quoted context omitted.

Indeed. It's almost as if testing should be built into the language itself. (I'm always a big fan of including self-test code in projects)

> testing should be built into the language itself I wonder what that would look like...

Rust has simple unit testing built into the language. And in Ada/Spark, tests can be a first class verification tool, alongside formal verification.

We should go a lot further though. IMO, a unit that does not pass a spec/test should cause a compile time error. Testing systems should facilitate and converge with formal verification. Where possible, property based testing should be used and encouraged. And debugger tools should be able to hone in on areas where the result diverges from expectations.

Re: The tragedy of 100% code coverage (2016)

#59

Post by cbanek »

saosebastiao wrote: post #58
thinkloop wrote: post #49

Earlier quoted context omitted.

> testing should be built into the language itself I wonder what that would look like...

Rust has simple unit testing built into the language. And in Ada/Spark, tests can be a first class verification tool, alongside formal verification. We should go a lot further though. IMO, a unit that does not pass a spec/test should cause a compile time error. Testing systems should facilitate and converge with formal verification. Where possible, property based testing should be used and encouraged. And debugger to…

^ This.

And another example of something I'd want checked by the language is exception throwing / handling. It's another one of those places where code coverage won't help you unless you already know what you're looking for. Languages are getting better about it, but in general, handling errors is hard.

Post reply on HN