Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Myth of Code Coverage

preslav.me

Post reply on HN
Print view View original

11–20 of 115 posts

Re: The Myth of Code Coverage

#11

Post by dahfizz »

There is a big difference here between strongly typed, compiled languages and weakly typed interpreted languages.

High code coverage is much more important on e.g. Javascript or Python because running the code is the only way you know that it "compiles" and you didn't do something dumb like mistake the type of arguments, access fields that don't exist, etc.

In a compiled codebase, I think functional / end-to-end tests are more important. I don't care as much if every single function is exercised by unit tests if I know that the software works as intended for all supported use cases.

Re: The Myth of Code Coverage

#12

Post by whymarrh »

g14i wrote: post #8
rzimmerman wrote: post #6

Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…

Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it. I highly recommend investing in mutation and fuzzy testing.

I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together.

  [1]:https://pitest.org

Re: The Myth of Code Coverage

#14

Post by leifg »

As with so many other code quality metrics and tools, automated testing (and by extension code coverage) is not necessarily about the number but about discovery.

If you are finding out that certain parts of your software are hard to test or will never be reached by any of your tests, that should tell you something about your architecture.

Re: The Myth of Code Coverage

#15

Post by dudul »

dahfizz wrote: post #11

There is a big difference here between strongly typed, compiled languages and weakly typed interpreted languages. High code coverage is much more important on e.g. Javascript or Python because running the code is the only way you know that it "compiles" and you didn't do something dumb like mistake the type of arguments, access fields that don't exist, etc. In a compiled codebase, I think functional / end-to-end test…

That's interesting because I think the exact opposite.

For dynamically typed languages I think it's more valuable to have end-to-end tests to make sure that the whole pipeline works correctly. I want to validate that all the call sites for function F are passing an int, as intended, as opposed to a string.

For statically typed languages, I already know that all call sites for F are passing an int. So I want to unit test all small parts of the pipeline, in isolation, to make sure I easily spot which part doesn't follow its contract. If a function F uses a function G for something, I'm gonna unit test F and make sure it does behave correctly for all possible returns from G, and that's it.

Re: The Myth of Code Coverage

#16

Post by morelisp »

hyperman1 wrote: post #7
ardy42 wrote: post #5

Earlier quoted context omitted.

> I was also going to say around 60-70 pct, but for a different reason: what's left in my code is mostly checking of assertions, debug logging and handling rare errors, i.e code that's not supposed to run. I grant you assertion-checking, but code for handling rare errors is not the code you should skip writing unit tests for. If it runs infrequently then you're far less likely to stumble on a regression during other…

Depends on rare errors. I meanthings like 'My database crashed halfway a transaction, my file system drops from under my application, my back end service gave up. You can't do much here. Dump some info, abrt the half- done work, maybe try again somewhere in the future. And yes, that last part migh deserve a test.

If you can't do much, it shouldn't be much code, so coverage should stay high.

I used to be of the "70% is good enough" school, but I found that as I wrote better programs, I also achieved better test coverage. Not because I was writing more tests, but because I was choosing core designs with fewer edge cases, simplifying my error handling with fewer reachable paths, and admitting I might as well just crash in more cases. So now my line/branch coverage is more like 95-100%, but my code is easier to read and I write _fewer_, mostly functional/behavioral, tests. Most lines I don't reach are fatal errors, and most fatal errors are the only statement in their branch.

Code you can't reach from test cases isn't a sign you should write more tests, it's a sign you should remove that code from the program.

Today I view 90-95% really as a baseline, and focus more on path coverage (most standard tools are quite bad for this still) and edge cases in data (e.g. denormed floats or different kinds of invalid data I want to make sure stay invalid) that don't affect coverage one way or another.

Re: The Myth of Code Coverage

#17

Post by g14i »

whymarrh wrote: post #12
g14i wrote: post #8

Earlier quoted context omitted.

Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it. I highly recommend investing in mutation and fuzzy testing.

I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together. [1]:https://pitest.org

whymarrh, great that you mentioned PIT. This is exactly what we use for Java as well.

For Python we use mutmut [1].

[1] - https://pypi.org/project/mutmut/

Re: The Myth of Code Coverage

#18

Post by Mindless2112 »

rzimmerman wrote: post #6

Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…

I like this parallelism between coverage and tests:

Coverage can prove that code is not tested but cannot prove that code is tested.

Tests can prove that bugs exist but cannot prove that bugs do not exist.

Re: The Myth of Code Coverage

#19

Post by DavidWoof »

Code coverage is an excellent tool but a terrible metric.

It's basically the lines-of-code of test driven development. It's useful as a general guide for developers, but its use as a management metric is extremely destructive.

Re: The Myth of Code Coverage

#20

Post by morelisp »

dudul wrote: post #15
dahfizz wrote: post #11

There is a big difference here between strongly typed, compiled languages and weakly typed interpreted languages. High code coverage is much more important on e.g. Javascript or Python because running the code is the only way you know that it "compiles" and you didn't do something dumb like mistake the type of arguments, access fields that don't exist, etc. In a compiled codebase, I think functional / end-to-end test…

That's interesting because I think the exact opposite. For dynamically typed languages I think it's more valuable to have end-to-end tests to make sure that the whole pipeline works correctly. I want to validate that all the call sites for function F are passing an int, as intended, as opposed to a string. For statically typed languages, I already know that all call sites for F are passing an int. So I want to unit t…

Typed language also make it possible to exhaustively test inputs. For functions that only have a few dozen or hundred possible inputs we can exhaustively test every single case (even millions/billions as part of a slow suite).

For looser languages, maybe the function takes a fuzzy notion of "a number" - OK, do I need to try the string "12"? What about "012"? What about "12.0" or " 12" or "\n12"?

Post reply on HN