Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

Post reply on HN
Print view View original

61–70 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#61

Post by throwawaylinux »

falcolas wrote: post #39
throwawaylinux wrote: post #23

Earlier quoted context omitted.

Not to disagree with the idea that you generally need more test cases than control flow paths to really test correctness well. Just a question about your example -- let's say your requirement is a function that does this: void fn(int a, int b) { if (a == b) printf("equal"); else printf("not); } What are the 3 test cases you would write? What are the 9? fn(1, 1) -> "equal" fn(1, 0) -> "not" What more useful tests are…

Let's say I mistype 'a == b' as 'a Let's add 1 to a before the equality comparison in order to meet a new business requirement. fn(1, 1) still works, but does fn(maxint, maxint)? Or does it suddenly throw an exception (or worse, silently roll over to minint)?

Right but you could also accidentally mistype it as 'b == 1'. Or 'a == b*b'. Or '(a == 0 || a == 1 || a == maxint) && a == b'.

I'm not saying more tests can't catch specific bugs you might come up with, I'm asking how you can choose numbers that have fundamental edge cases for this specific requirement without looking at the actual implementation. I don't think you really can. maxint/minint/0/-1/1 are generally common choices but only because they do tend to catch signdness errors or inequalities or division by zero or whatever, they aren't really fundamental to the requirement of this function though. If we just wanted to blindly blast those numbers in, it would be 25 individual test cases for all permutations. Doesn't seem to be reasonable.

Re: Coverage is not strongly correlated with test suite effectiveness

#62

Post by strken »

thaumasiotes wrote: post #45
tharkun__ wrote: post #10

In other words, more tests do find more bugs, but it's the number of tests and not their code coverage that has most of the predictive value. It's a surprising result, so if you'll excuse me, I have a couple of lecture slides on software testing I need to revise Is it just me or was this _not_ surprising at all? I mean I suppose I should have expected what he said, given it sometimes seems hard to convince other peop…

> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input? How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input. Now, if your code is just pumping data from one point to anoth…

This isn't true at all. Consider buffer overflows, bugs in your bitwise arithmetic, division by zero, or null pointer exceptions.

Re: Coverage is not strongly correlated with test suite effectiveness

#63

Post by tharkun__ »

fiddlerwoaroof wrote: post #47
tharkun__ wrote: post #41

Earlier quoted context omitted.

Very well said. I have nothing against TDD if that's something that helps a particular person write good code and good tests. I find though that people that write good tests are just people that write good tests. Most people write tests that assert on implementation details instead of inputs and output and that's easily doable via TDD as well.

It’s hard to assert based on implementation details if you don’t have an implementation yet: the advantage of test-first code is that you usually have to write the tests in terms of the interfaces of the input and output types rather than inspecting the code under test.

That's because you know how to write good tests but TDD didn't force you to do that.

Simply put, TDD just means you write the assertion first, see it red, then change the code to make it green.

You can do that by specifying a failing test via input/output testing. You can also very easily write a failing test that tests that the YXZ() method is gonna be called on some internal thing that you're replacing with a mock. Add that method and voila, green test, TDD happy. Bad test though.

Re: Coverage is not strongly correlated with test suite effectiveness

#65

Post by saalweachter »

sverhagen wrote: post #54

I've been asked many times what is the right code coverage percentage to aim for. I've also asked this many times in interviews, to tease out a discussion. My answer is: you need full coverage. I then continue to explain that I didn't say 100%. Although I believe that the average Java microservice (which is the general sphere that I move around) can easily achieve >98% coverage. Easily. But "full coverage" means that…

I've occasionally been lulled into a false sense of security by the difference between "every line is run" and "the function is fully tested".

On the other hand, even if I'm half-assing the unit tests, and implementing the bare minimum necessary to reach whatever arbitrary coverage% is being sought ... I still find bugs, bugs that almost certainly would have a real impact in production, even if I already ran an end-to-end test and was pretty sure it was working fine[*]. Not every time I write tests, but often enough to feel like it pays for all of the time "wasted".

So even if coverage metrics maybe arbitrary, I don't really begrudge having to meet them. And honestly, once you're forced to start writing the tests just for coverage, it isn't that much extra work to make them a little more comprehensive than the floor set by coverage.

[*] Honestly I'm shocked at how well buggy software works, for the most part.

Re: Coverage is not strongly correlated with test suite effectiveness

#66

Post by lwansbrough »

jeffbee wrote: post #57
lwansbrough wrote: post #48

Earlier quoted context omitted.

The notion that you can get software right the first time seems... a bit naive to me, regardless of the methodology, respectfully. We solved the internal API problems in our app simply by either versioning internal services/APIs (ie. MyServiceClassV2), adding a new method signature and deprecating the old one, or updating the existing method signature, which is pretty safe in strictly typed languages. (And if you wri…

I find your comments quite interesting. How do you validate MyServiceClassV2 if you didn't bother writing any tests for MyServiceClass? One of the benefits of testing is it enables refactoring. Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything.

It's not really about validation. Either MyServiceClass works or it doesn't (whether that's DX, or some other problem), and if it doesn't you make a new one or fix it. Most code isn't so complicated that coming up with the correct method signature is prohibitively difficult without writing tests.

If down the line the way it was written isn't working, maybe you need a V2. Anything important that needs updating to V2 gets updated, anything not worth the effort stays on V1. Usually that's not an issue. Sometimes you have to drop V1 for one reason or another, and that can result in some work, but I reckon it's time saved vs. writing test suites, and it always feels worthwhile to do, vs. writing insurance code.

> Without tests, code bases just become increasingly haunted graveyards where nobody is willing to change anything.

People will be fine with changing things if the cultural standard is that bugs can happen but just try to avoid them. As long as you have good logging it's largely a non-issue. I'd be more hesitant to change code if I had to update 10 tests along the way. Maybe that's laziness but it also feels wildly unproductive.

Re: Coverage is not strongly correlated with test suite effectiveness

#67

Post by vitus »

kragen wrote: post #59
jacobsenscott wrote: post #20

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…

I think it was Brian Marick who pointed out that the great benefit of a coverage report is that it tells you what you forgot to think about when you wrote the test suite. One response to code that isn't covered is to write the tests to exercise that code, but there are a couple of other possibilities he suggests might be better: 1. Can the uncovered code be removed from the system entirely? Maybe if none of the tests…

I find test-first most useful when debugging a hairy problem, and I think that's somewhere it's not used enough.

Reproducing the bug consistently* is the first step to understanding how it works, and how to fix it. And then you get the thrill of trying to turn that test green as you tinker.

...

* Okay, sometimes "consistently" is "fails about 1/100 executions", but that's not so bad if you can run your unit test 1000 times in the span of 30s.

Re: Coverage is not strongly correlated with test suite effectiveness

#68

Post by jrockway »

lwansbrough wrote: post #36

Now, I've never worked at a big company with lots of developers, but testing seems dramatically overvalued. My company currently employs zero testing (meaning zero automated tests). Anything that could negatively impact the company if it blew up is examined pretty closely and then set loose. Occasionally things break. Our company serves tens of millions of users monthly. We have bugs, we fix them as needed. We're not…

The problem is that you're missing guard rails. People know they have to be paranoid, so they move slowly; just like people drive more carefully without seatbelts and ride bicycles more carefully without helmets. Care is good, but does not come without cost.

I think it's hard to measure the impact of not testing, because in order to compare A vs B, you'll have to write tests (and it's a large job to write a test suite for existing code), but that's why it's insidious -- you know writing tests is going to slow you down, and you don't know how much time it's going to save in the long run. You end up with a heavy bias for the status quo. And, if f the tests aren't any good, you'll spend a lot of time writing them and you'll just get friction on future changes, slowing you down even more. But if the tests are good, then people that are new to the codebase can arrive with confidence.

Manual testing is good, but it scales O(n) with the number of features. What works for the simple prototype with two big features quickly becomes a drag when every engineer has to test all 100 features against every change they make. (That's why organizations make some other team do the tests, and then the bug comes back to you 3 weeks later, and then you stop what you're working on to fix the bug you caused -- context switching off of new feature work, and delaying that feature. Slow, and annoying!) The solution to this slowdown is usually more process (if we just write it down, it feels like it's not work) and "we need more engineers". The associated O(1) overhead of new proces plus the O(n!) communication slowdown means that the n in n features to test for every change grows much more slowly -- it's feels under control, but what you're really doing is less work with more resources.

I look at the tricky edge cases in my own work, and look at where the bugs creep in (we do a quarterly review of these), and it's always in the "that's too hard to test" code. Some examples: assuming that the test suite's view of the database is the same as a database without the new migrations applied, simple refactoring leading to null pointer dereferences in an edge case, third party applications that call into the API that "has no users" (according to grep), and things like that. These are the things that burn new team members, and make them overly cautious forever. Caution and velocity are incompatible, so to me, it's crucial that these dark corners get addressed, so a passing test suite means a codebase with only bugs we haven't seen before. (There are always going to be bugs, but you shouldn't fix the same bug twice.)

(Oh, and there are definitely bugs in code with 100% coverage. 100% coverage just means you found the bugs you already thought of, but it doesn't mean you found every bug. A test suite will never ensure your code is bug free.)

There are a lot of open source projects that lose their primary maintainer, and they never get another feature again because of this. Someone wants to add one, but they can't figure out how, and just rewrite it, or give up completely. Be on the lookout!

Re: Coverage is not strongly correlated with test suite effectiveness

#69

Post by hsn915 »

Aiming for 100% test coverage actually produces negative value.

You don't need a "study paper" to know this. Just work with a team that aims for 100% coverage for a few months and you will see it for yourself.

The negative value comes from:

1. The time wasted on writing all these tests that are mostly ceremonious in nature.

But, more importantly:

2. It makes refactoring a big pain in the ass.

Why? Because 100% test coverage forces you to test the implementation details of everything.

So, every time you do a non-trivial refactoring, you will get tens or hundreds of failing tests. This is mostly noise. If there's any signal in any of these failed tests (signal = information that you screwed something up during the refactoring) it will be very difficult to catch.

What do you do when you refactor something and break a 100 tests?

You delete all the failing tests and start again: look at the coverage report and start writing tests for the portions of the code that are not covered.

Post reply on HN