Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

231–240 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#231

Earlier quoted context omitted.

> testing should be built into the language itself I wonder what that would look like...

Rust has simple unit testing built into the language. And in Ada/Spark, tests can be a first class verification tool, alongside formal verification. We should go a lot further though. IMO, a unit that does not pass a spec/test should cause a compile time error. Testing systems should facilitate and converge with formal verification. Where possible, property based testing should be used and encouraged. And debugger to…

> IMO, a unit that does not pass a spec/test should cause a compile time error.

We can achieve this in dependently-typed languages like Idris. First we define a datatype 'Equal x y', which will represent the proposition that expression 'x' is equal to expression 'y':

    data Equal x y where
      Refl : (x : _) -> Equal x x
There are two things to note:

- There is only one way to construct a value of type `Equal x y`, which is to use the constructor we've called `Refl`.

- `Refl` only takes one argument, `x` (of arbitrary type `_`), and it only constructs values of type `Equal x x`. This is called "reflexivity", thus the name `Refl`.

Hence if we use the type `Expr x y` anywhere in our program, there is only one possible value we can provide that might typecheck (`Refl x`), and that will only typecheck if `Expr x x` unifies with `Expr x y`, which will only happen if `x` and `y` are the same thing; i.e. if they are equal. Thus the name `Equal`.

This `Equal` type comes in the standard library of all dependently typed languages, and is widely used. To use it for testing we just need to write a test, e.g. `myTest`, which returns some value indicating pass/fail, e.g. a `Boolean`. Then we can add the following to our program:

    myTestPasses : Equal myTest True
    myTestPasses = Refl myTest
This will only type-check if `Equal myTest myTest` (the type of `Refl myTest`) unifies with `Equal myTest True`, which will only be the case if `myTest` evaluates to `True`.

Re: The tragedy of 100% code coverage (2016)

#232
IMO, if I ever have 100% code coverage, I did something wrong. The best I can usually achieve is 95-98%, because of my defensive coding to warn about the "impossible" use cases.

Escape a `while True` loop? Log it, along with the current state of the program, and blow up (so we can be restarted). Memory allocation error? Log it. The big "unexpected exception" clause around my main function? Log it.

If I do hit those in testing, my code is wrong.

Re: The tragedy of 100% code coverage (2016)

#233
post #139

Earlier quoted context omitted.

If you absolutely want to cover all cases, you need to do mutation testing. A mutation testing system analyses your code, changes an operator (> becomes = for example), and then runs your tests. If one test fails, your tests covered that statement. It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim. The other option is t…

How would automated mutation testing handle the case of accidentally causing infinite loops, or invoking undefined behavior?

I use pitest to run mutation coverage on most of my Java code bases. pitest implements a timeout to check for infinite loops introduced by changing the code.

http://pitest.org/faq/

Re: The tragedy of 100% code coverage (2016)

#234

Earlier quoted context omitted.

> The worse the developer, the more tests he'll write. As always, generalization is the tool of the fool (sorry for the fool part, but it rhymes ;) ). Writing pointless stubs / mocks and testing execution order of statements is definitely a bad pattern, writing many and good functional, e2e and integration tests however is not.

Generalization is at the heart of science. The lack of generalization is one of the most frustrating attacks you can launch on a scientist's empiricism.

We are talking about software testing here.

To a first approximation, nobody is being empirical, and there is no science being done.

Re: The tragedy of 100% code coverage (2016)

#235

Earlier quoted context omitted.

If I am understanding correctly, he is saying that well-written code requires fewer tests in order to be fully tested. Fully tested is better than untested, assuming that code is fully tested, well written code will require fewer tests. That is my understanding, as it is rather hard to extract a solid argument from his posts.

Not writing tests is a great way to ensure future developers won't refactor your code

Oh I completely agree, and would much rather be working on a codebase with too many unit tests than too few. I'm trying to come up with the most charitable explaining that I can.

Re: The tragedy of 100% code coverage (2016)

#236

Laziness is a kind of populism. Such articles will be always upvoted.

He's not preaching laziness, he's against hitting metrics solely for the sake of hitting the metric when the metric is irrelevant. 100% code coverage? Good. 100% code coverage in a single test framework at 10 times the development cost when it's still not really 100%? Stupid. Working smarter isn't always in opposition to working harder.

No, he is exactly against covering simple code by tests. It's explained below first sentence.

  It is funny how things turn around. ​ For fifteen years I 
  have been preaching TDD (Test-driven development, or as it 
  used to be called: test-first approach), or at least for 
  developers to write some unit tests. However, in recent 
  times I have found myself saying more often, "Why did you 
  write that test?" instead of, "You should write a test."
Answer to that "funny turn around" sentence is simple - he is getting older and more lazy. Each of us will, each of us should fight with it.

And after that, example of what I'm talking about:

  It seems that he had trouble using Mockito to test the 
  following piece of code:
  ...
  I think he was very surprised with my response: "You 
  don't need to test that."

Re: The tragedy of 100% code coverage (2016)

#237

If you can prove that your testing process is perfect, then your entire development process can then be reduced to the following, after the test suite is written: cat /dev/random | ./build-inline.sh | ./test-inline.sh | tee ./src/blob.c && git commit -Am "I have no idea how this works, but I am certain that it works perfectly, see you all on Monday!" && git push production master --force When presented like this, rel…

Several other people are complaining about 100% coverage sometimes being misleading. One way to test your test is to randomly modify your code e.g. swapping a greater than for a less than. If your tests still pass, then they obviously missed this change in behaviour. This is known as mutation testing. https://en.wikipedia.org/wiki/Mutation_testing I've tried it a few times, I generally found it too slow to be an ever…

Have you ever tried it with one of the automated tools? I use Pitest extensively to mutate my Java code bases and found it is not too much slower then regular line and branch code coverage tools when you use a history file so only diffs are needed. The trade off of a few more seconds of build time is worth the benefit of the coverage report and being able to fail the build if coverage drops too low for me.

http://pitest.org/quickstart/maven/

Re: The tragedy of 100% code coverage (2016)

#238

Earlier quoted context omitted.

He's not preaching laziness, he's against hitting metrics solely for the sake of hitting the metric when the metric is irrelevant. 100% code coverage? Good. 100% code coverage in a single test framework at 10 times the development cost when it's still not really 100%? Stupid. Working smarter isn't always in opposition to working harder.

No, he is exactly against covering simple code by tests. It's explained below first sentence. It is funny how things turn around. ​ For fifteen years I have been preaching TDD (Test-driven development, or as it used to be called: test-first approach), or at least for developers to write some unit tests. However, in recent times I have found myself saying more often, "Why did you write that test?" instead of, "You sho…

A concise word for 'older and lazy' might be 'experienced'

Re: The tragedy of 100% code coverage (2016)

#239
Build tests against your app`s public interface. On a web app, that would be your controllers or API.

That will give you good coverage, while avoiding too simple to be useful unit tests.

It's really hard to foresee all possible input variations and business logic validations, but that doesn't mean your test suite is useless.

It just means it will grow everytime you find a new bug and you are guaranteed that one won't happen again...

Re: The tragedy of 100% code coverage (2016)

#240

Earlier quoted context omitted.

No, he is exactly against covering simple code by tests. It's explained below first sentence. It is funny how things turn around. ​ For fifteen years I have been preaching TDD (Test-driven development, or as it used to be called: test-first approach), or at least for developers to write some unit tests. However, in recent times I have found myself saying more often, "Why did you write that test?" instead of, "You sho…

A concise word for 'older and lazy' might be 'experienced'

Not always, not in this case definitely.
Post reply on HN