Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Coverage is not strongly correlated with test suite effectiveness

neverworkintheory.org

Post reply on HN
Print view View original

151–160 of 178 posts

Re: Coverage is not strongly correlated with test suite effectiveness

#151

Post by berkes »

hsn915 wrote: post #103
bronson wrote: post #87

Earlier quoted context omitted.

3. Your code base may start to become contorted. I've seen good programmers create bogus classes to allow test-time mocking, or add oddball env vars and configurations to let the test harness manually reach every last line. Even if that line is not worth testing: if(!(x=malloc(BUF_SIZ)) || ENV[TEST_MEM_FAIL_12]) { exit(1); } Tying code and tests this tightly discourages refactoring. Another example: a different code…

"Mocking" is such a weird thing to me and I don't think it serves a good purpose. It's the kind of thing that would only arise if you assume a-priori that 100% test coverage is a non-negotiable must. If you have a function A that calls B to get some data (by doing I/O) then process it using C, then the 100% cov rule would force you to mock B when you test A. But then what is the value of this test? What guarantees is…

Mocking helps with architecture, though.

When A calls C through B, you have a tight coupling between A and C, but mightn't be aware of that, because you only see A calling B.

Lets say C reads a CSV file. By testing implementation only, you might 'create a csv file' then call A.import() and assert some records in, say D are created. By testing A in isolation, you make apparent that it couples to C and D because you have to mock them out. B is a direct dependency of A, so you don't mock that. At the least, you have now documented the accidental coupling. But at the best, your tests showed you a design issue.

Re: Coverage is not strongly correlated with test suite effectiveness

#152

Post by regularfry »

brabel wrote: post #143
jacobsenscott wrote: post #134

Earlier quoted context omitted.

I hear this a lot, but the result is usually an untested, and usually nearly untestable (because it was written without tests), prototype with a few characterization tests that gets pushed to production and haunts you for the rest of the life of the product. Pototyping to define the problem or API is fine, but most people don't have the discipline to tear it out and start over when they finally do have a well defined…

> the result is usually an untested, and usually nearly untestable No way... if you ever wrote any tests, you can easily know how to write code that will be testable even if you do it later. Doing it before is just going to be a big waste of time if you throw the code away later, which happens a lot when you need to experiment with things before actually choosing what works best. Yes, you can do that with TDD as well…

Swings and roundabouts: what you lose on writing tests you gain on not bothering to implement things you don't need. If the test passes, you stop.

Without the tests to guide you it's very easy to waste time over-engineering, even if what you're building is well-structured.

Re: Coverage is not strongly correlated with test suite effectiveness

#153

Post by namelosw »

tpxl wrote: post #147
namelosw wrote: post #126

Testing is an art that takes quite a while to master. People are able to write tests but many don't know "how" to write tests. Testing is extremely valuable but at the same time very easy to get burned. When testing done wrong (like abusing tests for coverage): 1. It makes the code too inflexible to refactor, leaving the system too rigid to grow. 2. It makes people frustrated on how to write tests in this project. 3.…

> 1. Only test the behavior from the user/consumer's perspective, do not test implementations. (Therefore the internal can be refactored without deleting tests.) Sometimes you still need to test implementations. We had database writers/readers that automatically retried on failure/disconnect/failover/whatever and you can't test those by testing the API.

Thanks for pointing it out. I was not clear that I was referring to the tests related to business requirements (which is the majority of the tests).

Infrastructures could be treated like third-party modules as they could be tested like any open source project separately.

Re: Coverage is not strongly correlated with test suite effectiveness

#154

Post by shoo »

ParkerPierre wrote: post #145
satisfice wrote: post #71

Ever since I developed code coverage tools at Apple in 1989, and tested them for Borland in the early 90’s, I knew and have been telling people in MY conference slides that code coverage is a nearly useless metric. Anyone who thought critically about it for ten minutes knows it’s nonsense. The one thing code coverage tells you that is of any significant value is what you haven’t tested. You still know very little abo…

> Anyone who thought critically about it for ten minutes knows it’s nonsense Can you explain a bit more? I have only done coverage on my home projects. I get 100% every time because otherwise why bother. I started experimenting with 100% branch coverage. I understand sometimes there's a few lines you can't test like if there's a fork() but in my cases I was lucky enough to not need it and didn't need to exclude anyth…

> Can you explain a bit more?

Not the parent commenter, but i'll attempt to chime in: in addition to the default "dopey" practice of measuring coverage in terms of statements of code, in addition to data coverage as mentioned by the parent, you could also consider coverage in terms of possible execution paths through the code or coverage in terms of requirements. For interesting computational code (i dunno, consider a mixed integer program solver) it might be possible to get 100% statement or 100% branch coverage at the same time as getting approximately 0% data coverage, 0% execution path coverage, and 0% coverage of functional requirements.

Another perspective is how much value the test gives relative to the effort and cost to set it up and maintain it. You can have 100% unit test coverage (statement, branch, data, whatever) but maybe your application doesn't even do anything when it boots, because you forgot to call any of the units. So if effort and time is limited, you might get more return on investment from writing some good integration tests or hooking up a fuzzer than focusing on hitting the somewhat arbitrary but easy to measure 100% statement coverage number.

Also, like everything in life, focusing on a single objective like "statement coverage" and trying to maximise it can lead to strange outcomes. There was a fascinating blog post / war story somewhere describing a software system architecture where the organisation mandated non-negotiable minimum 80% statement coverage for unit test suites. This system had some pretty low level code and a bunch of that code wasn't possible to test in unit tests. So how do you hit the target? Well, if you add a few useless layers to your application architecture that are easy to unit test, then you test them and leave the hard code that interfaces with the real world / hardware untested, and you hit your mandated target, but the overall level of software quality is worse.

Re: Coverage is not strongly correlated with test suite effectiveness

#155

Post by thaumasiotes »

spankalee wrote: post #58
thaumasiotes wrote: post #45

Earlier quoted context omitted.

> Well you executed the branch/line at least once with one potential input. Was it an edge case input or a happy path input? How does that matter? If something about the input causes a difference in the execution of the code, then 100% coverage means you necessarily tests both kinds of input. You can't reach the edge case branch with the happy path input. Now, if your code is just pumping data from one point to anoth…

That's because you have such simple conditions. How do you compute them? if (x really changes things. Did you test for x === y? for NaN? for very close doubles that should have passed/failed for business reasons? For types other than numbers if your language allows? Expressions can have any number of edge cases that code coverage can't account for.

Well, imagine two different analyses of the code:

1. Does each branch do the right thing?

2. Does the code go down the correct branch in the first place?

I've focused more on the first question, and you're focusing more on the second. But it's fair to say that each question is a reasonable focus of testing, which weakens my comment above.

Re: Coverage is not strongly correlated with test suite effectiveness

#156

Post by amw-zero »

exdsq wrote: post #110
amw-zero wrote: post #98

Earlier quoted context omitted.

Yes I agree - I have been a testing fanatic for the better part of the last 10 years, after being absolutely paralyzed at a company without tests. But, after all this time, I believe their cost-to benefit-ratio is horrendous. It’s fairly common to hear of test suites with a 2:1 ratio of test to implementation lines. That would be fine if they didn’t immensely prevent refactoring and block merges / deployments. Contra…

The one thing is a spec doesn't mean the product is correct - you still need testing, just in a different way. It'll probably replace unit tests though.

Correct, there’s still nothing stopping us from writing the wrong spec or writing an insufficient spec, but there’s really no solution to that problem anyway. And the same problem exists with tests today - you can have an insufficient amount of test cases, or you can write the wrong cases because you misunderstand the requirements.

The difference being, when you realize your tests are wrong, what do you do? Change and add more test cases. You may also need to remove some, because they don’t make sense anymore.

Vs., updating logic in a specification, and that’s it.

Re: Coverage is not strongly correlated with test suite effectiveness

#157

Post by brabel »

regularfry wrote: post #152
brabel wrote: post #143

Earlier quoted context omitted.

> the result is usually an untested, and usually nearly untestable No way... if you ever wrote any tests, you can easily know how to write code that will be testable even if you do it later. Doing it before is just going to be a big waste of time if you throw the code away later, which happens a lot when you need to experiment with things before actually choosing what works best. Yes, you can do that with TDD as well…

Swings and roundabouts: what you lose on writing tests you gain on not bothering to implement things you don't need. If the test passes, you stop. Without the tests to guide you it's very easy to waste time over-engineering, even if what you're building is well-structured.

I often write code without implementing anything, just the "surface API". That's when I find whether things will work or not. Tests are a hindrance to that. Once I figure out the design, then I will test all that I think is important.

> what you lose on writing tests you gain on not bothering to implement things you don't need.

I am not sure where to start... I've written so much code, applications, libraries, algorithms... and I am pretty sure the opposite of what you say is true: with TDD, I would've spent hours trying to get something working that later I would find, by exploration, that I didn't need at all.

Re: Coverage is not strongly correlated with test suite effectiveness

#158

Post by amw-zero »

brabel wrote: post #146
amw-zero wrote: post #52

You don’t even need to do a study to arrive at this conclusion. Has no one read Djikstra’s writings? https://www.cs.utexas.edu/users/EWD/ewd02xx/EWD249.PDF “Program testing can be used to show the presence of bugs, but never to show their absence!” Everybody in our industry wants to bash math and say that anyone can write programs, but programs are logical systems and can only be fully understood with math and logic.…

> We have the tools for understanding and reasoning about infinitely large structures, programmers just refuse to use them, and even deride them. Could you point out which tools you're talking about?

Well, by this I mean math (and mathematical logic). Math is the tool for reasoning about possibly infinite concepts, i.e. you can make a statement about infinite sets and still know if it’s true or not. We aren’t limited to what we can see and touch, which is good because any non-trivial software application is so large that it could never be drawn out like a building blueprint. It can only be described and reasoned about abstractly.

But, you are probably asking about what ‘tools’ can be applied to programming, meaning some kind of library or application. These are also out there. Here’s a few that I think are promising:

TLA+: This is a specification language for describing and reasoning about computations as state machines, with a particular focus on modeling distributed systems. It has been used at Amazon to check designs for their distributed algorithms. They have used it to check parts of S3 for example: https://lamport.azurewebsites.net/tla/formal-methods-amazon....

Best reference is the book Specifying Systems: https://lamport.azurewebsites.net/tla/book-02-08-08.pdf.

Then you have theorem provers / proof assistants based on type theory. These are frankly complex, but getting better.

F*: This one is the most exciting to me. It’s currently being used to develop a formally verified HTTPS stack: https://project-everest.github.io/. They already have components released to the Linux kernel and Firefox. One of the most exciting things about this tool is that you can verify an efficient, stateful algorithm and extract highly performant C code from it. Their verified implementations have equal or better performance to the current solutions out there. So all of the overhead is for verification, none exists at runtime.

Of course you can’t talk about proof assistants without mentioning Coq. Its claim to fame is producing a formally verified C compiler, CompCert: https://compcert.org/.

As you can see, so far, formal verification has been mostly limited to components of systems, not entire systems. But, it’s a start, and the scope of what we can verify is getting larger.

Re: Coverage is not strongly correlated with test suite effectiveness

#159

Post by amw-zero »

Supermancho wrote: post #94
falcolas wrote: post #60

Earlier quoted context omitted.

Not all code is reachable. Especially things like top-level try/except clauses.

From the entry point, stub a function to throw an exception. You'll reach it. Unit testing isn't about finding every path. Nor is it about writing unit tests for every possible combination of values in a program. A java function that uses an int does not need to test -2147483648 to 2147483647 as inputs. That's not helpful.

That’s an interesting example, that’s exactly where you’re likely to find bugs - at boundary values. You should test those more than you should test the number 5.

This is exactly why unit testing gives a false sense of security. You’ve done a lot, you’ve written all kinds of tests - but at the end of the day, you don’t get a proportionate amount of confidence about the code because there are infinite more cases that you haven’t thought about.

Re: Coverage is not strongly correlated with test suite effectiveness

#160

Post by gilbetron »

exdsq wrote: post #115
gilbetron wrote: post #75

Say we have an application covered by 1000 tests. We fast forward a few years and (amuse this crazy notion) the application is decommissioned. We look and see that 500 tests never failed; those 500 always passed. Did we waste dev time by writing those tests? It's an interesting question to think about.

If I have a phone I need for work and insure it because if it breaks I could lose my client, but never actually need to claim on it, did I waste my money?

No, but I don't see the connection to my question. Maybe a slightly closer example is, if I have a phone, and 2 backup phones, and I never use the 2 backup phones, were they a waste?
Post reply on HN