Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

71–80 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#71

Post by BigJono »

ruleabidinguser wrote: post #53
mikestew wrote: post #28

The tragedy of 100% code coverage is that it's a poor ROI. One of things that stuck with me going on twenty years later is something from an IBM study that said 70% is where the biggest bang-for-the-buck is. Now maybe you might convince me that something like Ruby needs 100% coverage, and I'd agree with you since some typing errors (for example) are only going to come up at runtime. But a compiled (for some definitio…

70% coverage... I don't know if I'm going to be able to give up my 0% coverage 1-man projects...

I'd recommend aiming for 10-20% on those projects, and also for startups trying to rapidly push an MVP out.

Tests have diminishing returns. You want to hit the absolute most crucial ones that give you plenty of bang for buck and even save you time. That means finding the (usually small handful) of functions that implement your most crucial and most complicated business logic, and writing tests for them.

Anything past that is for companies with customers that need to maintain a certain level of quality and service. Worry about it when you get there.

Re: The tragedy of 100% code coverage (2016)

#73

Post by lobster_johnson »

cbanek wrote: post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

Code coverage should perhaps not be counted in lines, but in the number of type permutations covered. Then your challenge is determining all the types that _can_ be passed as inputs.

Re: The tragedy of 100% code coverage (2016)

#74

Post by raarts »

dcw303 wrote: post #69
eksemplar wrote: post #46

We've almost stopped unit testing. We still test functionality automatically before releasing anything into production, but we're not doing a unit test in most cases Our productivity is way up and our failure rates haven't changed. It's increased our time spent debugging, but not by as much as we had estimated that it would. I won't pretend that's a good decision for everyone. But I do think people take test-driven-d…

I'm curious - how old are the systems you're working on? In my experience, unit tests don't catch many bugs when the code is fresh. But when it's five years old with many modifications over the code base, some dumb little test that you thought was a waste of time is now alerting you to what would have been a horror regression. In other words: Even though it feels like it's slowing you down now, if you write tests whi…

Yeah good luck explaining those 5 years to the business manager.

Re: The tragedy of 100% code coverage (2016)

#75

Post by hashmp »

enraged_camel wrote: post #10
quantumhobbit wrote: post #6

Earlier quoted context omitted.

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…

Exactly. It's like testing a newly built bridge by crossing it a hundred times with a bike. Sure, you "covered" it, but you sure as hell did not test it.

Brilliant analogy.

Re: The tragedy of 100% code coverage (2016)

#76

Post by plinkplonk »

lacampbell wrote: post #41
jowiar wrote: post #38

Earlier quoted context omitted.

In Scala, "does it compile/typecheck" will generally catch typos in code that isn't frequently executed, and raise warnings for unhandled cases. For Ruby, you pretty much need to execute the code to have any inkling if it goes kaboom -- the language (and how people write it) have so much magic that short of executing things, the computer can't tell you anything useful.

Valid point. I don't think I could write low-bug count code faster in statically typed scala than I could in unit tested ruby though. I mean I am well aware of what you're telling me, and it's obvious to me that having the compiler automatically check certain properties is a win. And yet, when push comes to shove, to get something done I'm more likely to reach for ruby. It's something I've never come up with a good e…

"Does static typing stunt prototyping and exploration?"

Not in my experience. It is probably about what you personally find hard/uncomfortable/unfamiliar.

Re: The tragedy of 100% code coverage (2016)

#77

Post by johnwatson11218 »

Not sure if this is already mentioned but for me the most concise illustration of this fallacy was in The Pragmatic Programmer book. They had a function like this:

double f( double x ) { return 1/ x; }

They pointed out that it is trivial to get 100% coverage in test cases but unless your tests include passing in 0 as the parameter you are going to miss an error case.

Re: The tragedy of 100% code coverage (2016)

#78

Post by hgdsraj »

anamexis wrote: post #50
devrandomguy wrote: post #31

If you can prove that your testing process is perfect, then your entire development process can then be reduced to the following, after the test suite is written: cat /dev/random | ./build-inline.sh | ./test-inline.sh | tee ./src/blob.c && git commit -Am "I have no idea how this works, but I am certain that it works perfectly, see you all on Monday!" && git push production master --force When presented like this, rel…

It seems to me that such a "perfect" testing process would basically amount to declarative programming.

Well...SQL is declarative, and we still see "Select * from Users"

Re: The tragedy of 100% code coverage (2016)

#79

Post by hgdsraj »

ben_jones wrote: post #14
hibikir wrote: post #9

I might be completely wrong on this one, but it seems to me that a lot of the precepts of TDD and full code coverage have a lot to do with the tools that were used by some of the people that popularized this. Some of my day involves writing Ruby. I find using Ruby without 100% code coverage to be like handling a loaded gun: I can track many outages to things as silly as a typo in an error handling branch that went un…

IMO ruby is the worst because its so easy, and often times encouraged, to write layers of useless tests that ultimately give only a false sense of security. I recently re-did the Michael Hartl tutorial to catch up with Rails5 and it gives examples where you test proper template rendering. I love that book but I'd rather shoot myself in the foot then spend dev time writing tests to check if the right html title attrib…

If you can achieve >90% testing, efficiently and properly. It's worth it. Testing for the correct title is worth it. Especially if the title has something appended to it at render time (i.e. hello | website_name). Someone can over ride this on a page and having a test to catch that is important. A well written test would check for page_name | website_name across all pages. Pages are finite. Such tests are useful. Hartl uses basic testing framework as an introduction but there are better ones available.

Re: The tragedy of 100% code coverage (2016)

#80

Post by dullgiulio »

johnwatson11218 wrote: post #77

Not sure if this is already mentioned but for me the most concise illustration of this fallacy was in The Pragmatic Programmer book. They had a function like this: double f( double x ) { return 1/ x; } They pointed out that it is trivial to get 100% coverage in test cases but unless your tests include passing in 0 as the parameter you are going to miss an error case.

Exactly, or to put it another way: you are not forced to cover the code you didn't write; in this case, the non-zero check before the division.

Mix this with unchecked exceptions, mocking of any real-world interaction that will generate errors in productions, and testing becomes a cargo-cult of quality.

Post reply on HN