Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

121–130 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#121

Post by hultner »

Back when I learnt Haskell we had a lecturer named John Hughes who had co-authored a tool named QuickCheck[1]. We used this tool extensively throughout the course, with it testing were quite simple and writing elegant generators were a breeze. In my experience, these test did a much more greater job at finding edge cases then many unit tests I've seen in larger close to full coverage TDD projects.

As with much else TDD should be a tool with the ultimate goal of aiding us in writing correct and less bug riddled code, once the tool adds more work it's no longer offering much aid.

[1] . https://en.wikipedia.org/wiki/QuickCheck

Re: The tragedy of 100% code coverage (2016)

#122

Post by pjmlp »

sbov wrote: post #32
cbanek wrote: post #27

Earlier quoted context omitted.

> As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. One other thing about code coverage, unit testing, and other testing fads is I think they actively affect the architecture, and usually in the overthinking it way. Instead of having one tight bit of procedural code (which may have some state, or some dependency calls), peopl…

It's strange to me that we continue to make languages that force us to make certain architectural choices to help facilitate testing.

We have it since Eiffel and Common Lisp, but not all mainstream languages were keen on adopting design by contract.

On .NET it is a plain library, which requires the VS Ultimate editions to be useful and on Java it was mostly third party libraries.

C# design team is considering adding proper support as language feature, but it seems to be a very far away feature still.

C++20 might get contracts, but C++17 just got ratified, so who knows.

D, Ada 2012 and SPARK do already support contracts.

Re: The tragedy of 100% code coverage (2016)

#123

Post by dahauns »

UweSchmidt wrote: post #92
kasparsklavins wrote: post #88

Earlier quoted context omitted.

Tests are not meant for ensuring it works with all inputs. You cant simply just throw values at it hoping its all okay. To prove it works with all possible inputs, there are other tools at your disposal.

What tools would that be? I'd like to hear about real world test scenarios where all possible inputs are tested.

In practice, testing _all_ possible inputs is usually not a feasible goal, and often downright impossible (since it would mean solving the halting problem). While there are some languages and problem domains where formal verification methods are a possibility, for most of software development the best you can do is reducing solution space.

A sane type system would be a start for example. And it's no coincidence that functional programming and practices with emphasis on immutability are on the rise; Rusts ownership system is a direct consequence as well.

TDD _is_ important, if simply for enabling well-factored code and somewhat guarding against regression bugs. But - decades after Dijkstras statement (which someone has already posted in this thread) - the code coverage honeymoon finally seems to be over.

Re: The tragedy of 100% code coverage (2016)

#124

Post by dublinclontarf »

iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

I inherited a code base with LOTS of test coverage.

Made one small change and everything broke.

Looked into the tests, oh the humanity!!!

Dropped the test suite entirely, haven't had any issues since.

We have VERY good logging and notifications though so anything goes wrong we know all about it.

Re: The tragedy of 100% code coverage (2016)

#125

Post by dahauns »

ubernostrum wrote: post #90
cbanek wrote: post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

I only have one piece of code for which I can say true 100% coverage exists: a library that works with HTML/CSS color values, and which ships a test that generates all 16,777,216 hexadecimal and integer rgb() color values, and runs some functions with each value. However, I don't run that as part of the normal test suite. It only gets run when I'm prepping a new release, as a final verification step; the normal runs-…

Hmm...true 100% input coverage? Including the time domain? (Order dependency, Idempotence, etc...) :)

Re: The tragedy of 100% code coverage (2016)

#126

Post by pg314 »

The article illustrates what happens when you have inexperienced or poor developers following a management guideline.

To see how 100% coverage testing can lead to great results, have a look at the SQLite project [1].

In my experience, getting to 100% takes a bit of effort. But once you get there it has the advantage that you have a big incentive to keep it there. There is no way to rationalise that a new function doesn't need testing, because that would mess up the coverage. Going from 85% to 84% coverage is much easier to rationalise.

And of course 100% coverage doesn't mean that there are no bugs, but x% coverage means that 100-x% of the code is not even run by the tests. Do you really want your users to be the first ones to execute the code?

As an anecdote, in one project where I set the goal of 100% coverage, there was a bug in literally the last uncovered statement before getting to 100%.

[1] https://www.sqlite.org/testing.html

Re: The tragedy of 100% code coverage (2016)

#127

Post by BJanecke »

So generally, I write a test when I want to make an assumption an certainty. If I can't be certain that something is doing what it's supposed to I write a test for it, make sense?

So

``` Int => add(x, y) => x + y; ```

Doesn't get a test, however

``` Int => formulateIt(x, y) => (x * y)^y ```

Does

Re: The tragedy of 100% code coverage (2016)

#128

Post by arekkas »

iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

> The worse the developer, the more tests he'll write.

As always, generalization is the tool of the fool (sorry for the fool part, but it rhymes ;) ). Writing pointless stubs / mocks and testing execution order of statements is definitely a bad pattern, writing many and good functional, e2e and integration tests however is not.

Re: The tragedy of 100% code coverage (2016)

#129

Post by saltedmd5 »

The big error being made in this article (and most of the comments here) is the assumption that the purpose of unit tests is to "catch bugs." It isn't.

The purpose of unit tests is to document the intended behaviour of a unit/component (which is not necessarily a single function/method in isolation) in such a way that if someone comes along and makes a change that alters specified behaviour, they are aware that they have done so and prevented from shipping that change unless they consciously alter that specification.

And, if you are doing TDD, as a code structure/design aid. But that is tangential to the article.

Re: The tragedy of 100% code coverage (2016)

#130

Post by arekkas »

arekkas wrote: post #128
iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

> The worse the developer, the more tests he'll write. As always, generalization is the tool of the fool (sorry for the fool part, but it rhymes ;) ). Writing pointless stubs / mocks and testing execution order of statements is definitely a bad pattern, writing many and good functional, e2e and integration tests however is not.

On top of that this statement will lead (some) new developers to just say "oh yeah I don't need tests because the gosus write only a few tests as well". Same pattern can be observed in DOTA or other games where newbies say "oh I don't need wards/intel because pro players don't need it either". The difference is that pro players already know what's going on.

Also worth a read while we're at it: https://www.linkedin.com/pulse/20140623171038-114018241-test...

Post reply on HN