Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

311–320 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#311

Post by phlakaton »

algesten wrote: post #99
phlakaton wrote: post #61

Earlier quoted context omitted.

Some counterpoints: - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests. - If you're trying to figure out which part of a complicated system broke, having unit tests that break on the specific module, or class, or method can be quite helpful. - Yes, integration tes…

> - If you want to know if your utility classes and functions are sane, unit testing is far better bang for your buck than trying to figure out whether they're being adequately exercised in your service tests. I think we talk about the same thing, sometimes I will test something internal. I've thought about what I'm doing as considering larger "units" in my unit testing, but perhaps "functional testing" as parent int…

I'm kind of flexible on my terminology, perhaps more flexible than others, since my approach to testing has largely been self-taught from the experience of several shops.

To me, unit tests are when I'm testing the behavior of a "thing" in "isolation", for some definitions of those terms – and yes, I agree that those definitions vary greatly. However, I don't agree with you that the flexibility of the definition is an issue.

Integration testing for me is when I'm testing system interfaces, so I'm focusing on how my systems behave when they come together. Sometimes I do this in isolation, using stub services or even internal stubs to simulate another system's behavior, but generally I do it with an actual system when it's convenient.

I don't generally use the term functional testing because I personally think it's ambiguous – I'm testing functionality in either case! But I suspect our differences really just boil down to how you slice it. If you prefer to divide tests into black-box vs white-box, but don't care as much about the specific level of isolation involved in the test, functional testing is perhaps the term you'd prefer. I prefer to categorize tests in terms of the amount of isolation I'm using, in which case I'm basically thinking unit, integration, e2e.

Re: The tragedy of 100% code coverage (2016)

#312

Post by Nemcue »

lowbloodsugar wrote: post #287

I once joined a company that had 90% code coverage. After a while it became clear that there were all vanity tests: I could delete huge swathes of code with zero test failure. We let the contractors that wrote it move on, and we formed a solid team in house. We don't run code coverage any more because it makes the build run four times slower. Instead, I trust our teams to write the good tests. Sometimes that means So…

> If the test isn't necessary, then neither is the code.

How did you come to this conclusion?

What if the code is just glue code for different libraries?

Re: The tragedy of 100% code coverage (2016)

#313

Post by jdmichal »

tumblen wrote: post #242
BigJono wrote: post #71

Earlier quoted context omitted.

I'd recommend aiming for 10-20% on those projects, and also for startups trying to rapidly push an MVP out. Tests have diminishing returns. You want to hit the absolute most crucial ones that give you plenty of bang for buck and even save you time. That means finding the (usually small handful) of functions that implement your most crucial and most complicated business logic, and writing tests for them. Anything past…

How do you know what 10-20% to test? This sort of basic level of decision-making for testing is something I wish I had, but all the tutorials and guides are about 100% code-coverage TDD so it's hard to find a path to to learn reasonable, high ROI testing.

You should test the thing that makes you money first, and should delay testing supporting functionality. For instance, if I am writing a document converter, the thing that makes me money is the AST -> AST conversion. Testing that should come before testing parsing (bytes -> AST) and rendering (AST -> bytes).

The place where you make money is the place that will have the largest demand for new and changing functionality. And where things change the most is where you need tests to protect against regressions.

Re: The tragedy of 100% code coverage (2016)

#314

Post by sqeaky »

mikestew wrote: post #28

The tragedy of 100% code coverage is that it's a poor ROI. One of things that stuck with me going on twenty years later is something from an IBM study that said 70% is where the biggest bang-for-the-buck is. Now maybe you might convince me that something like Ruby needs 100% coverage, and I'd agree with you since some typing errors (for example) are only going to come up at runtime. But a compiled (for some definitio…

Doesn't it also depend on th application?

I mean I wouldn't settle for less than 200% test coverage on an automatic pilot for landing airplanes. If it is a one off script that happen to become a part of a temporary business process perhaps just a sample set of data, a desired output and small tool for comparing the results is enough.

Re: The tragedy of 100% code coverage (2016)

#315

Post by vram22 »

mannykannot wrote: post #308
vram22 wrote: post #302

Earlier quoted context omitted.

Partly related: Is equivalence partitioning [1] used much in the industry in software testing? [1] https://en.wikipedia.org/wiki/Equivalence_partitioning I came across the concept some years ago, IIRC, in the classic book The Art of Software Testing by Glenford Myers [2], and used it in a few projects. But have not really heard or read of it being used much, from people or on forums. [2] https://en.wikipedia.org/wiki…

In the form discussed in [1], it gets pretty complicated when more than one operand is a variable, and they are derived from prior operations on the original data. Alternatively, identifying equivalence partitions from the semantics of the input data faces the problem that a faulty program may create invalid partitions, so some test cases for a given equivalence partition pass, while others fail.

You raise some interesting points about issues with the method.

I had used the approach in your second paragraph above.

I agree that it can have the issue you mention. But is this not more or less the same as the issue that even test code can have bugs in it? But we still use test code. For that matter, even human testers doing manual testing can make mistakes. But we still do manual testing.

Re: The tragedy of 100% code coverage (2016)

#316

Post by sqeaky »

algesten wrote: post #30

My main issue with unit testing is what defines a unit? Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression. Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box. That…

> breaks whenever a minor internal detail changes.

This is usually where I draw the line for what to test. I don't care how a unit does what it does, I only care about what it does.

In my opinion a unit should have minimum side effects, none if possible, and the results should depend on the inputs. Write the test to the contract of what the caller should provide and caller will get back (and then any side that had to happening, like checking logs.)

Re: The tragedy of 100% code coverage (2016)

#317

Post by d6e »

ReligiousFlames wrote: post #64
cbanek wrote: post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

This is what I love about the design and operational philosophy of the Erlang/OTP evironment: the trickier class of harder-to-test/reproduce transient/timing pre-defensive-coding issues can sometimes be resolved automagically under supervision trees restarting said Erlang process. Log the stacktrace as a failure to maybe be fixed and move on. That's in addition to a robust soft-realtime, SMP VM which allows remote, l…

As a newbie to Erlang, how does Erlang handle this case better for a service than a Python process which logs and restarts on failure? Also, this runtime testing strategy doesn't handle cases where you have other systems depending on that Erlang code path being successful, does it?.

Re: The tragedy of 100% code coverage (2016)

#318

Post by penpapersw »

devrandomguy wrote: post #68
ruleabidinguser wrote: post #56

Earlier quoted context omitted.

Will they succeed?

Sure, as long as their users unanimously abide by a very large rulebook ;)

I'd say they are "succeeding" in the sense that they continue to make money. Whether they're truly successful in the sense of providing proper value to their customers to justify that money, and treating their programmers right, is another story, and I've seen enough cases where they don't do either of these, while the executives make plenty of money to live comfortably for years by exploiting both customers and their own programmers.

Re: The tragedy of 100% code coverage (2016)

#319

Post by mannykannot »

vram22 wrote: post #315
mannykannot wrote: post #308

Earlier quoted context omitted.

In the form discussed in [1], it gets pretty complicated when more than one operand is a variable, and they are derived from prior operations on the original data. Alternatively, identifying equivalence partitions from the semantics of the input data faces the problem that a faulty program may create invalid partitions, so some test cases for a given equivalence partition pass, while others fail.

You raise some interesting points about issues with the method. I had used the approach in your second paragraph above. I agree that it can have the issue you mention. But is this not more or less the same as the issue that even test code can have bugs in it? But we still use test code. For that matter, even human testers doing manual testing can make mistakes. But we still do manual testing.

I agree that it is a valid technique, and in fact it is widely used: a minimal case is when we use a test account ID to stand for all accounts. The pessimist in me was looking for the exceptions, which is actually not a bad trait when you are testing.

Re: The tragedy of 100% code coverage (2016)

#320

Post by joelthelion »

computerex wrote: post #282
joelthelion wrote: post #84

Earlier quoted context omitted.

That's why I'm highly skeptical when I hear the word "best practices". Sure, the intention is good, but it promotes mindless repeating of patterns over thinking about what really helps.

On the other hand, every discipline that I can think of has its own set of best practices. Why should software development be any different? I know a lot of people are susceptible to the mindless repeating, but that's not a fault of best practices.

I'm not against the idea per se, just highly skeptical when I hear it used, because more often then not, I've seen it used to promote mindless repetition of previously used patterns.
Post reply on HN