Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

181–190 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#181

Post by voidpointer »

ubernostrum wrote: post #90
cbanek wrote: post #18

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases. Just because you ran a function or ran a line doesn't mean it will work for the range of inputs…

I only have one piece of code for which I can say true 100% coverage exists: a library that works with HTML/CSS color values, and which ships a test that generates all 16,777,216 hexadecimal and integer rgb() color values, and runs some functions with each value. However, I don't run that as part of the normal test suite. It only gets run when I'm prepping a new release, as a final verification step; the normal runs-…

Just out of curiosity... What is the point in testing all 2^24 possible color values?

Re: The tragedy of 100% code coverage (2016)

#182

Post by koevet »

Retric wrote: post #178
koevet wrote: post #166

Earlier quoted context omitted.

> The worse the developer, the more tests he'll write. This is not only wrong, but also dangerous. I agree with the second part of your comment (about being pedantic about the number of unit tests), but a good and comprehensive test suite is fundamental for large projects, especially projects with many moving parts and a large turn-over of people (basically any big enterprise).

By far the best codebase I ever worked with had zero test cases, was nearly 30 years olds, been worked on by not just several teams but several companies, had basically zero documentation, and was still easy to read and understand. Unit tests only really catch a tiny fraction of bugs relating to defective code, not poor design, poor market fit, poor understanding of requirements, or any of the other bugs that most of…

Not sure where you are going with this story. Do you mean that tests are useless, because you once worked in a company that thought that a large, test-less codebase was a good idea?

Hint: anecdotal fallacy

Re: The tragedy of 100% code coverage (2016)

#183

Post by Retric »

koevet wrote: post #182
Retric wrote: post #178

Earlier quoted context omitted.

By far the best codebase I ever worked with had zero test cases, was nearly 30 years olds, been worked on by not just several teams but several companies, had basically zero documentation, and was still easy to read and understand. Unit tests only really catch a tiny fraction of bugs relating to defective code, not poor design, poor market fit, poor understanding of requirements, or any of the other bugs that most of…

Not sure where you are going with this story. Do you mean that tests are useless, because you once worked in a company that thought that a large, test-less codebase was a good idea? Hint: anecdotal fallacy

No, because I have worked with several dozen code bases, hundreds if not thousands of bugs, and Unit Tests are if anything a bad sign associated with code in desperate need of refactoring. They are the uncanny valley of quality.

Ed: Unit Tests become more valuable as code quality decreases.

Re: The tragedy of 100% code coverage (2016)

#184

Post by snarfy »

koevet wrote: post #166
iamleppert wrote: post #114

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

> The worse the developer, the more tests he'll write. This is not only wrong, but also dangerous. I agree with the second part of your comment (about being pedantic about the number of unit tests), but a good and comprehensive test suite is fundamental for large projects, especially projects with many moving parts and a large turn-over of people (basically any big enterprise).

Maybe large teams are worse at development than small teams, so they need more tests.

Re: The tragedy of 100% code coverage (2016)

#185

Post by _pmf_ »

UweSchmidt wrote: post #92
kasparsklavins wrote: post #88

Earlier quoted context omitted.

Tests are not meant for ensuring it works with all inputs. You cant simply just throw values at it hoping its all okay. To prove it works with all possible inputs, there are other tools at your disposal.

What tools would that be? I'd like to hear about real world test scenarios where all possible inputs are tested.

Polyspace [0] allows full state space checking. Usually, this is feasible because the kind of modules you test are heavily constrained 12 bit (or fewer) fixed point values (sensor output/ actor input) and contain no algebraic loops. With floats and/or large integers, this quickly becomes unwieldy (verification then takes weeks).

[0] https://de.mathworks.com/products/polyspace.html

Re: The tragedy of 100% code coverage (2016)

#186

Post by nuclx »

Kubuxu wrote: post #116
UweSchmidt wrote: post #92

Earlier quoted context omitted.

What tools would that be? I'd like to hear about real world test scenarios where all possible inputs are tested.

One of them would be a fuzz testing. You start with a predefined 'corpus' of inputs which are then modified by the fuzzer to reach yet not covered code paths. In the process it discovers many bugs and crashes. The input fuzzing process is rarely purely random. There are advanced techniques that allow the fuzzer to link input data to conditions of not covered branches. It is quite useful mechanism for checking inputs,…

Parent was talking about full coverage. While fuzzers are great tools, able to generate test inputs for easily coverable parts of the code, formally correct software has to be designed for provability.

Re: The tragedy of 100% code coverage (2016)

#187

Post by eksemplar »

dcw303 wrote: post #69
eksemplar wrote: post #46

We've almost stopped unit testing. We still test functionality automatically before releasing anything into production, but we're not doing a unit test in most cases Our productivity is way up and our failure rates haven't changed. It's increased our time spent debugging, but not by as much as we had estimated that it would. I won't pretend that's a good decision for everyone. But I do think people take test-driven-d…

I'm curious - how old are the systems you're working on? In my experience, unit tests don't catch many bugs when the code is fresh. But when it's five years old with many modifications over the code base, some dumb little test that you thought was a waste of time is now alerting you to what would have been a horror regression. In other words: Even though it feels like it's slowing you down now, if you write tests whi…

I work in the public sector in Scandinavia and some of our oldest systems still in service run on an old tandem. So some of it is pretty old.

This gives us some unique abilities in terms of modeling our productivity of course, because we started measuring before anyone thought up unit testing.

Over the past 15 years, unit testing has failed to produce anything positive, and test driven development has been an absolute disaster.

That being said, this isn't something which will be universally true. A lot of the software we use isn't build by us, and I'm certainly that a lot of the suppliers on that software use unit tests quite extensively.

For the things we build ourselves, however, there has been almost no value in adopting modern test philosophies.

You say that systems have to be able to be worked on 5 years from now, but the truth is that most of our systems transport data and rarely live 5 years without getting rewritten to deliver better performance, higher levels of security or simply because the business has changed completely. A lot of it hold very few responsibilities as well, making it extremely obvious what to fix when a service breaks.

Don't get me wrong, we've seen problems we wouldn't have with 100% coverage. But that doesn't matter when spending resources fixing them is still a net positive on every account.

Re: The tragedy of 100% code coverage (2016)

#188

Post by voidpointer »

johnwatson11218 wrote: post #77

Not sure if this is already mentioned but for me the most concise illustration of this fallacy was in The Pragmatic Programmer book. They had a function like this: double f( double x ) { return 1/ x; } They pointed out that it is trivial to get 100% coverage in test cases but unless your tests include passing in 0 as the parameter you are going to miss an error case.

passing 0 to f returns infinity (IEEE 754)

Not arguing the point here. It's just a terrible example.

Re: The tragedy of 100% code coverage (2016)

#189

Post by jt2190 »

Retric wrote: post #183
koevet wrote: post #182

Earlier quoted context omitted.

Not sure where you are going with this story. Do you mean that tests are useless, because you once worked in a company that thought that a large, test-less codebase was a good idea? Hint: anecdotal fallacy

No, because I have worked with several dozen code bases, hundreds if not thousands of bugs, and Unit Tests are if anything a bad sign associated with code in desperate need of refactoring. They are the uncanny valley of quality. Ed: Unit Tests become more valuable as code quality decreases.

Hmm... You're capitalizing "Unit Tests" like there's a need to distinguish the difference between those and what I infer is "code that was not Unit Tests that we ran to verify correctness". Am I correct? Are you drawing a distinction? Or did you truly have no code used to verify correctness? Or no automated way to test?

Re: The tragedy of 100% code coverage (2016)

#190

Post by mlvljr »

Retric wrote: post #178
koevet wrote: post #166

Earlier quoted context omitted.

> The worse the developer, the more tests he'll write. This is not only wrong, but also dangerous. I agree with the second part of your comment (about being pedantic about the number of unit tests), but a good and comprehensive test suite is fundamental for large projects, especially projects with many moving parts and a large turn-over of people (basically any big enterprise).

By far the best codebase I ever worked with had zero test cases, was nearly 30 years olds, been worked on by not just several teams but several companies, had basically zero documentation, and was still easy to read and understand. Unit tests only really catch a tiny fraction of bugs relating to defective code, not poor design, poor market fit, poor understanding of requirements, or any of the other bugs that most of…

Zero comments as well?
Post reply on HN