Coverage is not strongly correlated with test suite effectiveness • Hacker News Board

Re: Coverage is not strongly correlated with test suite effectiveness

#171

Post by exdsq » Wed, Sep 29, 2021, 1:54 AM UTC

logicchains wrote: post #133

exdsq wrote: post #110

Earlier quoted context omitted.

The one thing is a spec doesn't mean the product is correct - you still need testing, just in a different way. It'll probably replace unit tests though.

A spec can catch anything your tests can. And if your spec overlooks something, your tests will overlook it too.

I've seen projects using formal methods including mathematical specifications miss things caught in end-to-end tests written by domain experts rather than formal method engineers. Perhaps you're right in theory but I've yet to see it in practice.

Re: Coverage is not strongly correlated with test suite effectiveness

#172

Post by jbergens » Wed, Sep 29, 2021, 6:47 AM UTC

optymizer wrote: post #118

jacobsenscott wrote: post #20

It is easy to write a test that executes code without actually testing anything. I use coverage to find code with no tests all at, and write tests for that code. But once it is "covered" the coverage report is useless. In interpreted languages (ruby/python/etc) coverage at least tells you if there's a syntax error before running it in production, which is useful. Test first also improves the quality of the tests just…

I found TDD useful for a well defined problem or an agreed-upon API. For apps for example, especially those not well defined and designed as-you-go, where the designer and PM might change their minds frequently after toying around with the app or getting user feedback, TDD is a lot of overhead and tests after writing the code are primarily useful for preventing regressions when somebody else changes your code.

For prototyping TDD may be waste. If you are just trying different ux and different logic it may not help to test. For almost everything else I think it helps.

Re: Coverage is not strongly correlated with test suite effectiveness

#173

Post by satisfice » Thu, Sep 30, 2021, 8:32 AM UTC

ternaryoperator wrote: post #139

satisfice wrote: post #71

Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little abo…

> academic papers about testing are nearly useless. You must not have read the post--it's principally summary of a paper by two academics at the Univ. of Waterloo.

I did read the post, and I am referring to the paper by the academics. What did you think I was talking about? Hello is this mike on?

Re: Coverage is not strongly correlated with test suite effectiveness

#174

Post by satisfice » Thu, Sep 30, 2021, 8:33 AM UTC

shoo wrote: post #154

ParkerPierre wrote: post #145

Earlier quoted context omitted.

> Anyone who thought critically about it for ten minutes knows it’s nonsense Can you explain a bit more? I have only done coverage on my home projects. I get 100% every time because otherwise why bother. I started experimenting with 100% branch coverage. I understand sometimes there's a few lines you can't test like if there's a fork() but in my cases I was lucky enough to not need it and didn't need to exclude anyth…

> Can you explain a bit more? Not the parent commenter, but i'll attempt to chime in: in addition to the default "dopey" practice of measuring coverage in terms of statements of code, in addition to data coverage as mentioned by the parent, you could also consider coverage in terms of possible execution paths through the code or coverage in terms of requirements. For interesting computational code (i dunno, consider…

well said

Re: Coverage is not strongly correlated with test suite effectiveness

#175

Post by satisfice » Thu, Sep 30, 2021, 8:42 AM UTC

ParkerPierre wrote: post #145

satisfice wrote: post #71

Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little abo…

> Anyone who thought critically about it for ten minutes knows it’s nonsense Can you explain a bit more? I have only done coverage on my home projects. I get 100% every time because otherwise why bother. I started experimenting with 100% branch coverage. I understand sometimes there's a few lines you can't test like if there's a fork() but in my cases I was lucky enough to not need it and didn't need to exclude anyth…

Do you know what states are? Of course you do. So then you know that executing a line of code in one state could cause behavior that won’t occur in another state. Code coverage doesn’t say anything about state coverage.

You’ve heard of table lookups? Of course you have. So then you know that different results will come from a lookup that is based on a different query or hash key. The lookup is one line of code. But maybe you visit the code hundreds of times before you hit upon the result that gives you bad data which tanks the rest of your algorithm.

Come on, man. You know what I mean. Another example: I once wrote a table comparison routine in ObjectPal. I tested it and it seemed to work, but the first time I compared a table with embedded bitmaps I got “Object of type bitmap cannot be compared with operator”

See what I mean?

Re: Coverage is not strongly correlated with test suite effectiveness

#176

Post by satisfice » Thu, Sep 30, 2021, 8:51 AM UTC

amw-zero wrote: post #98

datavirtue wrote: post #96

Earlier quoted context omitted.

This is a good thing if you consider the price of not having them. I have worked in shops without tests. It's a great way to hand out free money to unsuspecting customers.

Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous. It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments. Contra…

I’m not sure what your point is about ratios. I don’t think they mean anything, strictly speaking. Perhaps they are related to some special knowledge you have that is not conveyed by the numbers.

Testing is a social act. Testing itself cannot be obsolete until complexity and misunderstanding is obsolete. This will happen never.

Perhaps certain kinds of shallow automated checking will become obsolete over time.

Re: Coverage is not strongly correlated with test suite effectiveness

#177

Post by amw-zero » Fri, Oct 01, 2021, 5:59 PM UTC

exdsq wrote: post #171

logicchains wrote: post #133

Earlier quoted context omitted.

A spec can catch anything your tests can. And if your spec overlooks something, your tests will overlook it too.

I've seen projects using formal methods including mathematical specifications miss things caught in end-to-end tests written by domain experts rather than formal method engineers. Perhaps you're right in theory but I've yet to see it in practice.

This just means that you thought of something in the test suite that wasn't in the spec. You can underspecify something.

Re: Coverage is not strongly correlated with test suite effectiveness

#178

Post by arwhatever » Sun, Oct 03, 2021, 3:50 AM UTC

malthuswaswrong wrote: post #112

arwhatever wrote: post #81

High coverage does not necessarily imply test suite effectiveness, nor does it even guarantee that your test suite provides positive value. But 0% coverage does tell you a lot about your test suite effectiveness. Also, beware the person too eager to proclaim that high test coverage is not necessarily good! Last time someone told me this was discussing process maturity during a job interview. I let the team slide with…

That's a shame. You could have gained new experiences and taught a team something new. Writing a program is easy and done a million times a day. Fixing a department is hard and people rarely get an opportunity to do it.

The Internet is a deluge of information and training material on this subject, so the teams that want to teach themselves this skill have already done so, leaving the teams that don’t want to do so.

I didn’t go into detail in my initial description, but this team misrepresented their process maturity to the extent that I had no qualms about bouncing out just as quickly as I got it to take a candid look around.

I’ve beaten my head against the wall for years on teams that didn’t give shit, so made testing and process maturity my primary focus during my job hunt, and made that clear to this team during the interview.