Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

271–280 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#271

Post by bpicolo »

mdpopescu wrote: post #267
bpicolo wrote: post #252

Earlier quoted context omitted.

> it makes absolutely no sense to test private members Don't agree. I think this is something that people tell themselves to justify the anti-pattern that is private members being nearly impossible to test directly in a variety of languages, but it seems like nonsense. I've never seen a real case for excluding private methods as a testable unit.

> I've never seen a real case for excluding private methods as a testable unit. Here's the reason: We're not writing tests; we're writing SPECIFICATIONS. Private methods are implementation details ; they're not part of the specification.

That's potentially one philosophy on it, sure. Yet that still doesn't preclude private methods from being an independently testable unit. Implementation details are the meat of the whole thing, and so they seem inherently testworthy. The glue code itself, less so.

Thanks for that thought though, definitely something I'll think on :)

Re: The tragedy of 100% code coverage (2016)

#272

Post by sqeaky »

iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

If we strike your first sentence, "The worse the developer, the more tests he'll write", then I can mostly agree with you.

It sounds like you have never worked on a project with no tests and those same developers that write "write long-winded, poorly abstracted, weird code that is prone to breaking". A test suite doesn't eliminate bad code or bad coders, but it does give the better people on the team a handle to grab them by.

Re: The tragedy of 100% code coverage (2016)

#273

Post by cowardlydragon »

KallDrexx wrote: post #216
cbanek wrote: post #27

Earlier quoted context omitted.

> As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. One other thing about code coverage, unit testing, and other testing fads is I think they actively affect the architecture, and usually in the overthinking it way. Instead of having one tight bit of procedural code (which may have some state, or some dependency calls), peopl…

I think there's an argument to be made that if the desire for testing is a main driver for your architecture then your tests are too granular, and you aren't testing any internal integration points. In my experience that means that your tests are so tied to your current architecture that you can't even refactor without having to rewrite tests.

mocking / interaction / expect breaks encapsulation to perform "testing".

Thus it is often a test of the implementation's assumptions when first written, and even worse, when the code is maintained/edited, the test is merely changed to get it to pass, because unit tests with mocks are usually:

1) fragile to implementation 2) opaque as to intent

Whereas input/output integration points are more reliable, transparent, and less fragile to implementation changes if the interface is maintained.

However, if you must do mock-level interaction testing, Spock has made it almost palatable in Javaland.

This is one area where functional fans get to make the imperative folks eat their lunch.

Re: The tragedy of 100% code coverage (2016)

#274

Post by computerex »

rjdavis3 wrote: post #233
MereInterest wrote: post #180

Earlier quoted context omitted.

How would automated mutation testing handle the case of accidentally causing infinite loops, or invoking undefined behavior?

I use pitest to run mutation coverage on most of my Java code bases. pitest implements a timeout to check for infinite loops introduced by changing the code. http://pitest.org/faq/

Pitest has solved the halting problem?

Re: The tragedy of 100% code coverage (2016)

#275

Post by sqeaky »

Retric wrote: post #178
koevet wrote: post #166

Earlier quoted context omitted.

> The worse the developer, the more tests he'll write. This is not only wrong, but also dangerous. I agree with the second part of your comment (about being pedantic about the number of unit tests), but a good and comprehensive test suite is fundamental for large projects, especially projects with many moving parts and a large turn-over of people (basically any big enterprise).

By far the best codebase I ever worked with had zero test cases, was nearly 30 years olds, been worked on by not just several teams but several companies, had basically zero documentation, and was still easy to read and understand. Unit tests only really catch a tiny fraction of bugs relating to defective code, not poor design, poor market fit, poor understanding of requirements, or any of the other bugs that most of…

Survivorship bias.

What about all the shitty 30 year code bases with no tests and no documentation that got dropped because they were garbage.

It's not impossible to write good code without tests, it is just more likely that code with tests and docs will be good enough to survive 30 years. Your project with no tests or docs and was good might be 1 in a million, while software that is good with tests and docs might be 1 in 100.

Re: The tragedy of 100% code coverage (2016)

#276

Post by mcv »

mannykannot wrote: post #260
mcv wrote: post #139

Earlier quoted context omitted.

If you absolutely want to cover all cases, you need to do mutation testing. A mutation testing system analyses your code, changes an operator (> becomes = for example), and then runs your tests. If one test fails, your tests covered that statement. It seems to me you need to have serious OCD to go for 100% mutation coverage, but that is what you really need to do if perfect coverage is your aim. The other option is t…

There can be circumstances in which the substituted operator is just as valid as the operator being substituted for, resulting in false positives.

How would that be a false positive? If the substituted operator does not cause a test to fail, then your tests don't cover it. If the function of the program is not changed by changing the operator, then the operator does nothing and should be removed.

That's the idea at least. All focus on high coverage, whether line coverage or mutation testing, creates an incentive to remove redundant robustness checks, and maybe that's not such a good idea after all. But that's a problem with all unit testing, and not just with mutation testing.

Re: The tragedy of 100% code coverage (2016)

#277

Post by mcv »

computerex wrote: post #274
rjdavis3 wrote: post #233

Earlier quoted context omitted.

I use pitest to run mutation coverage on most of my Java code bases. pitest implements a timeout to check for infinite loops introduced by changing the code. http://pitest.org/faq/

Pitest has solved the halting problem?

Timeouts are not a perfect solution to the halting problem, but usually good enough.

Re: The tragedy of 100% code coverage (2016)

#278

Post by stephencanon »

StavrosK wrote: post #145
Gibbon1 wrote: post #135

Earlier quoted context omitted.

I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible. Brute force. Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results. Me I have a pi…

How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.

With floating-point math libraries especially, it's frequently the case that it's relatively easy to compute the desired result accurately by using higher precision for internal computations; e.g. you might compare against the same function evaluated in double and then rounded to float.

This sounds pretty sketchy at first, but it turns out that almost all of the difficulty (and hence almost all of the bugs) are in getting the last few bits right, and the last few bits of a much-higher precision result don't matter when you're testing a lower-precision implementation, so combined with some basic sanity checks this is actually a very reasonable testing strategy (and widely used in industry).

Re: The tragedy of 100% code coverage (2016)

#279

Post by slavik81 »

jameshart wrote: post #207
StavrosK wrote: post #145

Earlier quoted context omitted.

How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.

One answer to that is to apply property based testing. If you're testing an addition algorithm, for example, you might test that A + B == B + A A + 0 == A (A + B) + C == A + (B + C) A + B != A + C (given B != C)

It's a good idea in principle, though your third and fourth properties are not true of floating point addition.

When you have one available, testing against the results against a reference implementation is a little more direct. Though, given how easy those property tests are, you might as well do both.

Re: The tragedy of 100% code coverage (2016)

#280

Post by bluGill »

voidpointer wrote: post #188
johnwatson11218 wrote: post #77

Not sure if this is already mentioned but for me the most concise illustration of this fallacy was in The Pragmatic Programmer book. They had a function like this: double f( double x ) { return 1/ x; } They pointed out that it is trivial to get 100% coverage in test cases but unless your tests include passing in 0 as the parameter you are going to miss an error case.

passing 0 to f returns infinity (IEEE 754) Not arguing the point here. It's just a terrible example.

You are both correct and wrong: I want X/0 to immediately crash (fatal error) before the bad data that caused my to try to divide by zero gets propagated farther. Now that I think about it, I really want X/Y where Y is "close" to 0 to crash too.

Of course I made the above up on the spot, but it is a reasonable thing to do for some situations. Those who know floating point math are well aware that dividing by something close to zero tends to result in very large errors, vs the true answer. (particularly if the division is part of a larger calculation)

Post reply on HN