Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

171–180 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#171

Post by jacques_chester »

mcv wrote: post #139
cbanek wrote: post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

If you absolutely want to cover all cases, you need to do mutation testing. A mutation testing system analyses your code, changes an operator (> becomes = for example), and then runs your tests. If one test fails, your tests covered that statement. It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim. The other option is t…

Strictly, mutation testing can't assure all cases either -- combinatorial state explosions are fun. But it does improve assurance a lot.

Re: The tragedy of 100% code coverage (2016)

#172

Post by praptak »

StavrosK wrote: post #145
Gibbon1 wrote: post #135

Earlier quoted context omitted.

I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible. Brute force. Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results. Me I have a pi…

How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.

If a function takes a single 32 bit argument you can run it on every possible value.

Re: The tragedy of 100% code coverage (2016)

#173

Post by jacques_chester »

kasparsklavins wrote: post #120
Kubuxu wrote: post #116

Earlier quoted context omitted.

One of them would be a fuzz testing. You start with a predefined 'corpus' of inputs which are then modified by the fuzzer to reach yet not covered code paths. In the process it discovers many bugs and crashes. The input fuzzing process is rarely purely random. There are advanced techniques that allow the fuzzer to link input data to conditions of not covered branches. It is quite useful mechanism for checking inputs,…

Fuzzing is essentially the same as just throwing values at it hoping its all okay . What I meant was what @AstralStorm said about mathematical and logic proofs that it works for all defined values.

Fuzz testing is useful at testing whole systems in vivo.

Proofs are powerful, but you can still make errors in the proof or the transcription into code. And of course you don't know what other emergent behaviours will surprise you when the proved code interacts with unproved code.

Depending on your means and needs, you want to try both.

Re: The tragedy of 100% code coverage (2016)

#174

Post by detaro »

praptak wrote: post #172
StavrosK wrote: post #145

Earlier quoted context omitted.

How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.

If a function takes a single 32 bit argument you can run it on every possible value.

… this reply does nothing to answer the parent's question. It's obvious that you can run it on all possible values, but that doesn't answer the question of the return values. (unless you only want to verify "doesn't crash" or something similarly basic)

Re: The tragedy of 100% code coverage (2016)

#175

Post by MereInterest »

scottschulthess wrote: post #168
mianos wrote: post #161

Earlier quoted context omitted.

I wish to inherit code that is clear, concise and works over any tests thanks. Each to his own.

So you're saying test coverage is a negative? It's not an either or.

If I am understanding correctly, he is saying that well-written code requires fewer tests in order to be fully tested. Fully tested is better than untested, assuming that code is fully tested, well written code will require fewer tests.

That is my understanding, as it is rather hard to extract a solid argument from his posts.

Re: The tragedy of 100% code coverage (2016)

#176

Post by detaro »

StavrosK wrote: post #145
Gibbon1 wrote: post #135

Earlier quoted context omitted.

I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible. Brute force. Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results. Me I have a pi…

How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.

Your comparison source could be running on a different platform, or be slower, or ... and thus not be a drop-in replacement. (E.g. in the example of a broken floating-point unit, compare the results with a software emulation)

Re: The tragedy of 100% code coverage (2016)

#177

Post by EugeneOZ »

iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

The worse the developer, the more they sure they can write perfect code which doesn't need any tests because it's perfect and will be perfect even after future changes, even after changes in related code - absolutely perfect code.

Re: The tragedy of 100% code coverage (2016)

#178

Post by Retric »

koevet wrote: post #166
iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

> The worse the developer, the more tests he'll write. This is not only wrong, but also dangerous. I agree with the second part of your comment (about being pedantic about the number of unit tests), but a good and comprehensive test suite is fundamental for large projects, especially projects with many moving parts and a large turn-over of people (basically any big enterprise).

By far the best codebase I ever worked with had zero test cases, was nearly 30 years olds, been worked on by not just several teams but several companies, had basically zero documentation, and was still easy to read and understand.

Unit tests only really catch a tiny fraction of bugs relating to defective code, not poor design, poor market fit, poor understanding of requirements, or any of the other bugs that most often make it into production. They can be a huge benefit to many possibly most projects, but there are no silver bullets, good quality needs competent people given sufficient time and resources and nothing really changes that equation.

Re: The tragedy of 100% code coverage (2016)

#180

Post by MereInterest »

mcv wrote: post #139
cbanek wrote: post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

If you absolutely want to cover all cases, you need to do mutation testing. A mutation testing system analyses your code, changes an operator (> becomes = for example), and then runs your tests. If one test fails, your tests covered that statement. It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim. The other option is t…

How would automated mutation testing handle the case of accidentally causing infinite loops, or invoking undefined behavior?
Post reply on HN