Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

331–340 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#331

Post by ubernostrum »

test1235 wrote: post #115
ubernostrum wrote: post #90

Earlier quoted context omitted.

I only have one piece of code for which I can say true 100% coverage exists: a library that works with HTML/CSS color values, and which ships a test that generates all 16,777,216 hexadecimal and integer rgb() color values, and runs some functions with each value. However, I don't run that as part of the normal test suite. It only gets run when I'm prepping a new release, as a final verification step; the normal runs-…

How long does that take?

Depends on the machine. See the comments in the file:

https://github.com/ubernostrum/webcolors/blob/master/tests/f...

The fun part is people who criticize the generation of the integer triplets; yes, it's three nested loops and that's bad, but the total number of iterations will always be 16,777,216 no matter what algorithm you decide to use to generate them. So it uses nested loops since that's the most readable way to do it.

Re: The tragedy of 100% code coverage (2016)

#332

Post by tracker1 »

cbanek wrote: post #27
pacaro wrote: post #25

Earlier quoted context omitted.

This was one of the fun things about working on Midori, the whole system was available to us, components all communicated through well defined interfaces, you really could (if necessary) mock the interfaces above and below and be sure that your logic was being exercised. The browser team, in particular, pushed hard to be close to 100% coverage. The overall number for the project was (IIRC) in the 85% range. When I'm…

> As an aside, while I'm rambling, all the examples in the article appeared to represent unnecessary abstraction, which is the opposite problem. One other thing about code coverage, unit testing, and other testing fads is I think they actively affect the architecture, and usually in the overthinking it way. Instead of having one tight bit of procedural code (which may have some state, or some dependency calls), peopl…

I think that depends on the language/environment... for example, it's usually MUCH easier to get high coverage, and to do just enough mocking testing against Node-style modules in JS than logic in a typical C# or Java project.

Usually because interfaces need to be clearly/well defined and overridden... In JS there are tools (rewire, proxyquire, etc) that can be used with tests in order to easily inject/replace dependencies without having to write code to support DI nearly as much. In fact, I'd say that there's usually no reason not to be very close to 100% coverage in JS projects.

Re: The tragedy of 100% code coverage (2016)

#333

Post by tracker1 »

realusername wrote: post #292
vincnetas wrote: post #47

Earlier quoted context omitted.

What are the languages that have testing not as afterthought?

Definitely Ruby in general, testing is a core of the language culture.

I think that's the nature of a dynamic/scripted language... it's just easier to handle in Ruby and JS than many other languages.

Re: The tragedy of 100% code coverage (2016)

#334

Post by sobani »

tumblen wrote: post #242
BigJono wrote: post #71

Earlier quoted context omitted.

I'd recommend aiming for 10-20% on those projects, and also for startups trying to rapidly push an MVP out. Tests have diminishing returns. You want to hit the absolute most crucial ones that give you plenty of bang for buck and even save you time. That means finding the (usually small handful) of functions that implement your most crucial and most complicated business logic, and writing tests for them. Anything past…

How do you know what 10-20% to test? This sort of basic level of decision-making for testing is something I wish I had, but all the tutorials and guides are about 100% code-coverage TDD so it's hard to find a path to to learn reasonable, high ROI testing.

Uncertainty. Look for the parts of the code where you feel you least sure you have the logic right.

A simple example of this is the bowling game kata: given the throws in a bowling game, calculate the (final) score. The 'hard' part is to keep strikes and spares in mind, including the bonus throws when a strike or spare is scored in the 10th frame.

If you were making an application that would help you keep track of your bowling score, that score calculator would have the highest ROI in terms of testing.

Re: The tragedy of 100% code coverage (2016)

#335

Post by josteink »

> The tragedy is that once a "good practice" becomes mainstream we seem to forget how it came to be, what its benefits are, and most importantly, what the cost of using it is.

Totally agree. You can say this about lots of things really and not just tests.

Re: The tragedy of 100% code coverage (2016)

#336

Post by saltedmd5 »

defined wrote: post #51
quantumhobbit wrote: post #6

Earlier quoted context omitted.

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…

> The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. This is true. But isn't that true of any test? A suite of naive or badly written tests can also give us a false sense of security, so why write any at all (I don't mean that literally)? I think that a greater level of confidence in our code can be ach…

> If it's a trivial line of code like a getter or setter that cannot possibly be wrong, then it should be fairly trivial to generate automatically a test case for it.

More importantly, if it's a getter or setter, the actual behaviour to be tested is probably broader than "should set/get x."

Test scenarios, not methods.

Re: The tragedy of 100% code coverage (2016)

#337

Post by mcv »

mannykannot wrote: post #285
mcv wrote: post #276

Earlier quoted context omitted.

How would that be a false positive? If the substituted operator does not cause a test to fail, then your tests don't cover it. If the function of the program is not changed by changing the operator, then the operator does nothing and should be removed. That's the idea at least. All focus on high coverage, whether line coverage or mutation testing, creates an incentive to remove redundant robustness checks, and maybe…

I was thinking that, for example, that a >= test is as valid as == in some cases. If == is actually used and is correct, substituting >= would not cause any valid test to fail. As I wrote this, however, I realized that if you substitute the inverse operator (!= for ==, = etc.) and it is not caught, you can infer it is not covered. (edit: or that, in the context of the program as a whole, the result of the operation i…

Substituting == for >= should cause a test to fail. Either == is correct, or >=. They can't both be correct. Should the two things always be equal? Then they shouldn't be unequal. Is one allowed to be bigger than the other? Then that should be allowed, and not rejected.

If this change doesn't cause a test to fail, you're not testing edge cases for this comparison.

(Of course that's assuming you want perfect coverage.)

Re: The tragedy of 100% code coverage (2016)

#338

Post by camus2 »

bpicolo wrote: post #271
mdpopescu wrote: post #267

Earlier quoted context omitted.

> I've never seen a real case for excluding private methods as a testable unit. Here's the reason: We're not writing tests; we're writing SPECIFICATIONS. Private methods are implementation details ; they're not part of the specification.

That's potentially one philosophy on it, sure. Yet that still doesn't preclude private methods from being an independently testable unit. Implementation details are the meat of the whole thing, and so they seem inherently testworthy. The glue code itself, less so. Thanks for that thought though, definitely something I'll think on :)

> That's potentially one philosophy on it, sure.

That's not potentially one philosophy, that's the definition of unit testing.

> Yet that still doesn't preclude private methods from being an independently testable unit

Then these methods ain't private as they break encapsulation. You can't have it both ways. Either a method is private or it isn't.

> Implementation details are the meat of the whole thing

But unit testing isn't about testing implementation details, it's about testing a specification that your API must respect, because that's the behavior collaborators consuming that API rely on. If your collaborators can call a private API then that API isn't private at first place.

Re: The tragedy of 100% code coverage (2016)

#339

Post by collyw »

shusson wrote: post #72

Has anyone read any papers about the relationship between code coverage and defects?

Not exactly what you asked for but this compares defects and different types of testing. Unit testing is one of the less effective ways of finding bugs.

https://kev.inburke.com/kevin/the-best-ways-to-find-bugs-in-...

Re: The tragedy of 100% code coverage (2016)

#340

Post by 0hjc »

AstralStorm wrote: post #100
gontard wrote: post #97

Earlier quoted context omitted.

I think that mutation testing should replace code coverage. It really ensure that the test verify the code behavior and not only invoked it.

The trouble with mutation testing is that most mutations will completely break an application. Why not prove the code is correct instead? Should be much cheaper than 100% coverage and more certain.

Why do you say most mutations will completely break an application? This is certainly not my experience.

Mutation testing systems normal use fairly stable operators (e.g changing a > to a >=). In most locations in the code changes such as these will have only a subtle effect.

Post reply on HN