Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

71–80 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#71
Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense.

The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little about what you have tested.

Meanwhile, this article continues the dopey practice of using the word “coverage” only with respect to code and not with respect to many other forms of coverage, such as data. Data coverage is not easy to measure, yet it is far more important.

Generally speaking, academics know very little about testing, and academic papers about testing are nearly useless. If you want to learn about testing from an academic, read sociology, philosophy, and cognitive science.

Re: Coverage is not strongly correlated with test suite effectiveness

#72
post #69

Aiming for 100% test coverage actually produces negative value. You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage for a few months and you will see it for yourself. The negative value comes from: 1. The time wasted on writing all these tests that are mostly ceremonious in nature. But, more importantly: 2. It makes refactoring a big pain in the ass. Why? Because 100% test c…

It is very unpopular view.

I have never been a fan of unit testing and instead I prefer end to end functional tests -- where you write tests that verify your application still behaves exactly as expected but does not care how it is implemented.

This usually requires much less code, does not deter refactoring and also focuses on the one thing that is really important for the client.

Since the outside interface of the application is less likely to change compared to the implementation, the tests tend to be more stable and require less maintenance over life of the application.

Unit tests were supposed to help refactoring (by making it easier to ensure the code still works after change). The sad reality is that I can never trust unit tests. I still need to research the code around the function I am modifying to be sure I am not breaking anything.

Also my personal style is to do shit ton of refactoring. This means I usually start with something that only vaguely resembles the end result. I spend a lot of time moving stuff around until I get rid of everything that I don't like.

This absolutely precludes writing unit tests up front. But also, after I have spend so much time polishing the code, writing unit tests for it is absolutely the last thing I want to do. There is absolutely no possibility that people will put effort into writing tests well if they did good job writing the actual implementation.

Yet another reason why unit tests are broken is that there is no mechanism, no feedback loop to ensure quality of tests.

When you write application code, if it is broken you will have feedback in the form of defects and outages or various other problems.

One reason that documentation is usually broken is because there typically does not exist mechanism that would ensure that documents stay in sync with design and implementation.

The same problem with unit tests. The only strong signal you get is when the test does not work. But there is no good signal to fix real important problems like missing tests for important part of the contract.

Re: Coverage is not strongly correlated with test suite effectiveness

#73

Earlier quoted context omitted.

Let's say I mistype 'a == b' as 'a Let's add 1 to a before the equality comparison in order to meet a new business requirement. fn(1, 1) still works, but does fn(maxint, maxint)? Or does it suddenly throw an exception (or worse, silently roll over to minint)?

Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'. I'm not saying more tests can't catch specific bugs you might come up with, I'm asking how you can choose numbers that have fundamental edge cases for this specific requirement without looking at the actual implementation. I don't think you really can. maxint/minint/0/-1/1 are generally commo…

Is it reasonable? It depends on your goals. Do you want to ensure that, for the set of well known edge cases, your function is correct?

Proving the function is correct for the known edge cases for integers and integer comparison is not excessively hard. Let's go for the worst case of edge cases, and:

- Pick the input pairs that are problems with ints: the set of (minint, maxint).

- Pick the input pairs that are problems with comparisons: the set of (-1, 0, 1).

- Pick input pairs that show that it's correct where a b. That last set of cases is nicely covered by the previous two.

That's a total of 25 input pairs(5^2). In my opinion, that's not an unreasonable number of test cases to test for correctness. Why do I think it's reasonable? Because while it's technically 25 test cases, it's really just one test that runs through an array of input values in a loop. Really simple, and adding more test cases as the business logic changes is equally simple.

And let's be honest, given how quickly computers work these days, even testing all ints for both inputs isn't completely unreasonable (though I wouldn't make it a part of a unit test suite). Especially if your logic is more complicated than a simple equality test (which most function logic will be).

EDIT: "Accidentally mistype" 'a==b' as '(a == 0 || a == 1 || a == maxint) && a == b'

Love it.

Re: Coverage is not strongly correlated with test suite effectiveness

#74
post #37

Earlier quoted context omitted.

I was going to be worried if mutation testing wasn't mentioned here. Is a great way to test the effectiveness of your tests at catching the common mistakes people make in code. That is, a mutation suit doesn't test your code, per se. It tests your test suit.

I've worked a lot on Java, and while there is great mutation testing tools (well, PIT specifically), I find them hard to "scale" practically, i.e. to run them in an automated fashion "every time". You either get long running builds with them in it, or you have to deal with the logistics of moving caches around (so that the mutation testing can be incremental). And as just another tool that a developer MAY use if they…

No, I found the same. Team disabled them on every release build, as they felt it was too slow. I may bring them back in the release servers, but it is honestly trying to argue for these when nobody else cares.

Re: Coverage is not strongly correlated with test suite effectiveness

#75
Say we have an application covered by 1000 tests. We fast forward a few years and (amuse this crazy notion) the application is decommissioned. We look and see that 500 tests never failed; those 500 always passed. Did we waste dev time by writing those tests?

It's an interesting question to think about.

Re: Coverage is not strongly correlated with test suite effectiveness

#76
post #67
post #59

Earlier quoted context omitted.

I think it was Brian Marick who pointed out that the great benefit of a coverage report is that it tells you what you forgot to think about when you wrote the test suite. One response to code that isn't covered is to write the tests to exercise that code, but there are a couple of other possibilities he suggests might be better: 1. Can the uncovered code be removed from the system entirely? Maybe if none of the tests…

I find test-first most useful when debugging a hairy problem, and I think that's somewhere it's not used enough. Reproducing the bug consistently* is the first step to understanding how it works, and how to fix it. And then you get the thrill of trying to turn that test green as you tinker. ... * Okay, sometimes "consistently" is "fails about 1/100 executions", but that's not so bad if you can run your unit test 1000…

Oh, yeah, that's a good point; even people who don't do test-first most of the time will often do it for bug fixes.

I think the importance of performance to software quality via automated testing is profoundly underestimated.

Re: Coverage is not strongly correlated with test suite effectiveness

#77

Earlier quoted context omitted.

They go hand in hand. It’s obvious that missing branch coverage means your data cases are not exercising all of the edge cases in the code.

Not all code is reachable. Especially things like top-level try/except clauses.

They're in your source code - they are reachable.

Re: Coverage is not strongly correlated with test suite effectiveness

#78

Earlier quoted context omitted.

Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'. I'm not saying more tests can't catch specific bugs you might come up with, I'm asking how you can choose numbers that have fundamental edge cases for this specific requirement without looking at the actual implementation. I don't think you really can. maxint/minint/0/-1/1 are generally commo…

Is it reasonable? It depends on your goals. Do you want to ensure that, for the set of well known edge cases, your function is correct? Proving the function is correct for the known edge cases for integers and integer comparison is not excessively hard. Let's go for the worst case of edge cases, and: - Pick the input pairs that are problems with ints: the set of (minint, maxint). - Pick the input pairs that are probl…

I'm not sure that it is reasonable. It doesn't scale up to anything but the most trivial functions. It also does not prove the function is "actually correct", that is a fallacy. If you want that, you have to use formal.

The problem is not this one particular function, it's transferring this methodology to something more complex. A reasonable set of tests that does not aim to prove correctness is quite entitled to assume a reasonable implementation. You wouldn't have to arbitrarily re-run all your test cases with values incremented by 42 just in case someone incorrectly subtracted 42 somewhere, for example.

Re: Coverage is not strongly correlated with test suite effectiveness

#80

Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little abo…

Yet it's extremely popular to have huge, inefficient test suites running on continuous integration servers hundreds of times a day. It boggles the mind.
Post reply on HN