Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

201–210 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#201

Post by saltedmd5 »

quantumhobbit wrote: post #6
ioquatix wrote: post #3

I have 100% code coverage on a couple of projects. It has two benefits: Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected. New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the…

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…

This is completely true, but it's an argument against using coverage as a metric for test quality, not against test coverage per se.

Re: The tragedy of 100% code coverage (2016)

#202

Post by jcadam »

Retric wrote: post #178
koevet wrote: post #166

Earlier quoted context omitted.

> The worse the developer, the more tests he'll write. This is not only wrong, but also dangerous. I agree with the second part of your comment (about being pedantic about the number of unit tests), but a good and comprehensive test suite is fundamental for large projects, especially projects with many moving parts and a large turn-over of people (basically any big enterprise).

By far the best codebase I ever worked with had zero test cases, was nearly 30 years olds, been worked on by not just several teams but several companies, had basically zero documentation, and was still easy to read and understand. Unit tests only really catch a tiny fraction of bugs relating to defective code, not poor design, poor market fit, poor understanding of requirements, or any of the other bugs that most of…

I worked on one such system (written in Ada, of all things) back when I was a fairly junior developer. I was hired to work on a major effort to port the system from SPARC/Solaris to RHEL/x86 and add some new features. It was designed by a crusty (and eccentric) old architect (who was still around to manage this major refresh project) who ruled with an iron fist, and the project was managed in classic waterfall fashion.

He was not afraid to look you in the eye and say: "Your crappy code is not going into my codebase until you fix X, Y, Z, and change your indents to match the style guide." There were no "unit tests", per se, only a test driver program that he had written himself.

He was a hard guy to like, but I had to admit, that system had a clean, coherent, and modular design the likes of which I haven't seen since.

Re: The tragedy of 100% code coverage (2016)

#203

Post by ZeroGravitas »

devrandomguy wrote: post #31

If you can prove that your testing process is perfect, then your entire development process can then be reduced to the following, after the test suite is written: cat /dev/random | ./build-inline.sh | ./test-inline.sh | tee ./src/blob.c && git commit -Am "I have no idea how this works, but I am certain that it works perfectly, see you all on Monday!" && git push production master --force When presented like this, rel…

Several other people are complaining about 100% coverage sometimes being misleading. One way to test your test is to randomly modify your code e.g. swapping a greater than for a less than. If your tests still pass, then they obviously missed this change in behaviour.

This is known as mutation testing.

https://en.wikipedia.org/wiki/Mutation_testing

I've tried it a few times, I generally found it too slow to be an everyday part of my routine, but interesting tool to have in your belt for e.g. evaluating test in a codebase you didn't write yourself.

Re: The tragedy of 100% code coverage (2016)

#204

Post by threatofrain »

iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

Any causal phenomena has some number of factors which it depends upon. A test ought to provide large insight to the user with respect to which factors are the most likely culprits to a specific expression of the phenomena, and they also ought help a programmer determine the number of factors at play.

If a competent programmer cannot quickly determine how many factors are in play, or if that programmer cannot constrain the number of relevant factors to a humanly manageable number, then I question how insightful the sum of tests were.

Re: The tragedy of 100% code coverage (2016)

#205

Post by ZeroGravitas »

dragonwriter wrote: post #103
WalterBright wrote: post #101

Of course any metric can be rendered useless if one "works the metric" rather than the intent of the metric. But in my experience, code covering unit tests have correlated strongly with faster development and far fewer bugs being uncovered in the field.

> Of course any metric can be rendered useless if one "works the metric" rather than the intent of the metric. If the metric actually measures the figure of merit rather than a proxy with no necessary linkage, this is not true; working the metric then is working the intent of the metric. The extent to which a metric can be "worked" distinct from its intent is exactly the extent to which it is measuring something disc…

Even if the measure is accurate, there's an opportunity cost for spending time on things that are also important, but not measured. You'd need a measure that covered everything, from sufficient speed optimisations to fulfilling customer desires to complying with all relevant legal requirements, which basically can't exist.

Re: The tragedy of 100% code coverage (2016)

#206

Post by mirko22 »

The title does not mean anything and is basically a click bait in my opinion as it sounds cool to trash some ideal. That said, the magic 100% number is far removed from reality and does not represent anything by itself.

100% coverage on project of which size? Imagine you have a single script project that does exactly one thing and 2 test are enough to verify that it works without doing it manually? That is not the same as writing test for file system or tests which consists mostly of mocks upon mocks upon mocks.

I think the real problem is someone comes up with an idea, like TDD, tells people about it, some people hear about, start preaching it, some people start believing it and nobody actually think things through, usually cos they don't have experience (it's not a fetish as someone said). Like everything in life, you have to think things through before doing them, ask your self is this worth doing and when it is worth doing. You can't just say: "Oh we are doing TDD thus everything must be done in TDD way".

For people that say tests are useless, or good code does not need tests, I ask, when you make a change do you still make sure your code works by hand? And if you do make sure, why don't you automate that? You are a programmer after all.

And for those that say you need to test everything, well you don't, specially if you need mock most of it or it is really not that important piece of code as it is dev tool or something. What you want to make sure works is customer/user facing stuff that must work for you to get paid and you want to be able to verify this at any time of day without losing hours clicking around checking for stuff.

So this is not straight forward, 100% means nothing without context and doing anything in excess and without valid reasons is pointless or even harmful. And this has nothing to do with programming but life in general.

Re: The tragedy of 100% code coverage (2016)

#207

Post by jameshart »

StavrosK wrote: post #145
Gibbon1 wrote: post #135

Earlier quoted context omitted.

I'm reminded of a guy tasked with testing a 32 bit floating point library. After a number of false starts he realized the most effective way possible. Brute force. Oh other story. An OG (original geek) I know once proved the floating point unit on a mainframe was broken via brute force. Bad part meant the decimal floats used by and only by the accounting department sometimes produced erroneous results. Me I have a pi…

How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.

One answer to that is to apply property based testing. If you're testing an addition algorithm, for example, you might test that

   A + B == B + A
   A + 0 == A
   (A + B) + C == A + (B + C)
   A + B != A + C (given B != C)

Re: The tragedy of 100% code coverage (2016)

#208

Post by mianos »

scottschulthess wrote: post #168
mianos wrote: post #161

Earlier quoted context omitted.

I wish to inherit code that is clear, concise and works over any tests thanks. Each to his own.

So you're saying test coverage is a negative? It's not an either or.

Test coverage is a plus but not more important. We have an end to end test system that runs on each commit. What I am saying is perfect code, simply written is way way more important than complete coverage. Tests, on my experience, don't make code better written or more concise. Given a choice between coverage and good code I choose good code. In my experience time is always short so I choose the time on good code. In practice this is a compromise anywhere but the most cashed up companies.

Re: The tragedy of 100% code coverage (2016)

#209

Post by pg314 »

thehardsphere wrote: post #155
pg314 wrote: post #126

The article illustrates what happens when you have inexperienced or poor developers following a management guideline. To see how 100% coverage testing can lead to great results, have a look at the SQLite project [1]. In my experience, getting to 100% takes a bit of effort. But once you get there it has the advantage that you have a big incentive to keep it there. There is no way to rationalise that a new function doe…

Doesn't what you're writing actually influence whether 100% coverage is a worthy goal or not? I mean, SQLite is a good example of something where 100% coverage would actually be useful, because it tries to maintain compatibility with the SQL spec and with Postgres (largely because Postgres complies with the spec). Testing that 100% makes a lot of sense. Suppose you're instead doing something very UX/UI driven. Why bo…

> Doesn't what you're writing actually influence whether 100% coverage is a worthy goal or not?

Yes, absolutely.

> Suppose you're instead doing something very UX/UI driven. Why bother trying to cover that 100% with automated tests, when change is going to be driven by the whim and fancy of anyone who sits down in front of it?

It might make perfect sense from a business standpoint to have no tests at all.

Re: The tragedy of 100% code coverage (2016)

#210

Post by ZeroGravitas »

elchief wrote: post #55

What you actually want to do is test the methods with the highest cyclomatic complexity first (where it's greater than 1) IntelliJ has a plugin

Most standard unit coverage tools will do this for you, often with the CRAP (Change-risk anti-patterns) score:

http://www.ncover.com/blog/change-risk-anti-patterns-code-co...

"This is based on the methodology that the more complex code is the more likely it is to have errors, hence the need for greater code coverage.

To better understand, let’s assume a scenario where you wanted to keep your change risk anti-patterns score below 30. To maintain this level with a code set that has a cyclomatic complexity of 10, you would need to achieve a code coverage (as defined by branch coverage) ratio of approximately 42%. If however, the complexity of your code was greater and you had a cyclomatic complexity of 20, you would need almost 72% code coverage if you wanted to maintain the same risk level.

Complexity increases risk. Testing decreases risk. The change risk anti-patterns score give you a metric to measure the correlation between the two."

Post reply on HN