Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

131–140 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#131

Post by xg15 »

I agree (mostly) with the authors standpoints, but his arguments to get there are not convincing:

> You don't need to test that. [...] The code is obvious. There are no conditionals, no loops, no transformations, nothing. The code is just a little bit of plain old glue code.

The code invokes a user-passed callback to register another callback and specifies some internal logic if that callback is invoked. I personally don't find that obvious at all.

Others may find it obvious. That's why I think, if you start with the notion "this is necessary to test, that isn't", you need to define some objective criteria when things should be tested. Relying on your own gut feeling (or expecting that everyone else magically has the same gut feeling) is not a good strategy.

If I rewrite some java code from vanilla loops-with-conditionals into a stream/filter/map/collect chain, that might make it more obvious, but it wouldn't suddenly remove the need to test it, would it?

>"But without a test, anybody can come, make a change and break the code!"

>"Look, if that imaginary evil/clueless developer comes and breaks that simple code, what do you think he will do if a related unit test breaks? He will just delete it."

You could make that argument against any kind of automated test. So should we get rid of all kinds of testing?

Besides, the argument doesn't even make sense. No one is using tests as a security feature against "evil" developers (I hope). (One of) the points of tests is to be a safeguard for anyone (including yourself) who might change the code in the future and might not be aware of all the implications of that change. In that scenario, it's very likely you change the code but will have a good look at the failed test before deciding what to do.

Re: The tragedy of 100% code coverage (2016)

#132

Post by ocharles »

hultner wrote: post #121

Back when I learnt Haskell we had a lecturer named John Hughes who had co-authored a tool named QuickCheck[1]. We used this tool extensively throughout the course, with it testing were quite simple and writing elegant generators were a breeze. In my experience, these test did a much more greater job at finding edge cases then many unit tests I've seen in larger close to full coverage TDD projects. As with much else T…

Also see

https://github.com/hedgehogqa https://hackage.haskell.org/package/smallcheck

Re: The tragedy of 100% code coverage (2016)

#133

Post by antirez »

Code coverage is an illusion, since what you want is actually "possible states coverage". You can cover all the lines of your code and still cover a minority of the possible states of the program, and especially a minority of the most probable states of the program when the actual users execute it, or you can cover 50% of your lines of code and yet cover much more real world states, and states for which it is more likely to find a bug. I think that more than stressing single features with unit tests, it is more useful to write higher level stress tests (fuzz tests, basically) that have the effect of testing lines of code as a side effect of exploring many states of the program. Specific unit tests are still useful, but mostly in order to ensure that edge cases and main normal behavior corresponds to the specification. As in everything it is develooper's sensibility that should drive what test to write.

Re: The tragedy of 100% code coverage (2016)

#135

Post by Gibbon1 »

kasparsklavins wrote: post #88
cbanek wrote: post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

Tests are not meant for ensuring it works with all inputs. You cant simply just throw values at it hoping its all okay. To prove it works with all possible inputs, there are other tools at your disposal.

I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible.

Brute force.

Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results.

Me I have a pitiful amount of unit tests for the main codebase I work on. One module which does have unit tests, also had a bug that resulted in us having to replace a couple of thousand units in the field.

Otherwise most of the bugs that chap my keister aren't about procedural correctness, they're about state.

Re: The tragedy of 100% code coverage (2016)

#136

Post by camus2 »

iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

> The worse the developer, the more tests he'll write.

Spare us your platitudes. Most developers don't decide the code coverage, the manager or the client does.

Re: The tragedy of 100% code coverage (2016)

#137

Post by dahauns »

saltedmd5 wrote: post #129

The big error being made in this article (and most of the comments here) is the assumption that the purpose of unit tests is to "catch bugs." It isn't. The purpose of unit tests is to document the intended behaviour of a unit/component (which is not necessarily a single function/method in isolation) in such a way that if someone comes along and makes a change that alters specified behaviour, they are aware that they…

The big error being made in this article (and most of the comments here) is the assumption that the purpose of unit tests is to "catch bugs."

More like "in countless articles, comments, talks and projects over, say, the last decade".

Re: The tragedy of 100% code coverage (2016)

#138

Post by camus2 »

ardit33 wrote: post #45
algesten wrote: post #30

My main issue with unit testing is what defines a unit? Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression. Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box. That…

I agree with you. That's called functional testing, and it is very useful, but it is not unit testing. Unit testing: Test all methods and paths of a class, even private ones. Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests. ROI of functional testing is high, as it is usually done with real data. In m…

> even private ones.

You don't test private API, that's why they are declared private, it makes absolutely no sense to test private members.

Now the use of the private member might lead to a different code path : it has to be tested.

Unit Testing : test one unit ( a class for instance ) in isolation, which means all collaborators (the classes the tested class depends on) have to be stubbed or mocked.

Functional Testing : test multiple unit at the same time to check the behavior of an entire functionality, that's why it is called functional testing. Stub the IO.

End-to-end testing: Test the entire system with the IO.

Re: The tragedy of 100% code coverage (2016)

#139

Post by mcv »

cbanek wrote: post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

If you absolutely want to cover all cases, you need to do mutation testing. A mutation testing system analyses your code, changes an operator (> becomes = for example), and then runs your tests. If one test fails, your tests covered that statement.

It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim.

The other option is to accept that perfect coverage is not feasible, or possibly even a trap, and that you just need to cover the interesting bits.

Re: The tragedy of 100% code coverage (2016)

#140

Post by scottschulthess »

iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

So the worst of us developers out there write tons of tests? We wish. Who cares if it's 80% or 100%, i'd rather inherit a codebase with lots of test coverage than without.
Post reply on HN