Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Myth of Code Coverage

preslav.me

Post reply on HN
Print view View original

91–100 of 115 posts

Re: The Myth of Code Coverage

#91

Post by voodootrucker »

rzimmerman wrote: post #6

Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…

You can hit 100% with zero assertions if you like.

Probably not the test you want to have though.

Re: The Myth of Code Coverage

#92

Post by regularfry »

voodootrucker wrote: post #91
rzimmerman wrote: post #6

Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…

You can hit 100% with zero assertions if you like. Probably not the test you want to have though.

Zero assertions means that you know there aren't any lurking exceptions triggered by just running the code. That's a good thing to know, but I'd hope like hell that it's not the only thing you want to know.

Re: The Myth of Code Coverage

#93

Post by joshschreuder »

regularfry wrote: post #92
voodootrucker wrote: post #91

Earlier quoted context omitted.

You can hit 100% with zero assertions if you like. Probably not the test you want to have though.

Zero assertions means that you know there aren't any lurking exceptions triggered by just running the code. That's a good thing to know, but I'd hope like hell that it's not the only thing you want to know.

It may not even prove that depending on your mock/fake strategy though.

Re: The Myth of Code Coverage

#94

Post by boxed »

Mindless2112 wrote: post #18
rzimmerman wrote: post #6

Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…

I like this parallelism between coverage and tests: Coverage can prove that code is not tested but cannot prove that code is tested. Tests can prove that bugs exist but cannot prove that bugs do not exist.

Mutation testing can prove that the tests check all the behavior of the code, but it can't prove that the code shouldn't have more behavior, or that this behavior is correct.

Re: The Myth of Code Coverage

#95

Post by dalke »

nullc wrote: post #76
dalke wrote: post #43

Earlier quoted context omitted.

I wrote a UDF for MySQL. It had just shy of 100% coverage. It used a malloc. I could not figure out how to trigger a malloc failure inside of a imported library running on a MySQL instance. Do I not check for malloc failure (in order to get 100%)? Or do I develop some sort of instrumented version of MySQL which lets me test malloc failure? That seemed far more overkill than warranted for the small project I worked on…

> Could not figure out how to trigger a malloc failure inside of a imported library Glibc has (or at least has had) special (unsupported) instrumentation that lets you substitute out the libc malloc at runtime with another function. You can use this to shim in a version that fails after a specific number of executions in order to inject malloc faults. E.g. https://github.com/xiph/opus/blob/master/tests/test_opus_api.…

I considered using a shim, but couldn't figure out which malloc comes from my UDF and which comes from MySQL, and I was highly doubtful that MySQL would have identical numbers of malloc by the time my code runs.

Re: The Myth of Code Coverage

#96

Post by dalke »

mathgladiator wrote: post #57
dalke wrote: post #43

Earlier quoted context omitted.

I wrote a UDF for MySQL. It had just shy of 100% coverage. It used a malloc. I could not figure out how to trigger a malloc failure inside of a imported library running on a MySQL instance. Do I not check for malloc failure (in order to get 100%)? Or do I develop some sort of instrumented version of MySQL which lets me test malloc failure? That seemed far more overkill than warranted for the small project I worked on…

A core cultural challenge that we have to deal with is that people are very soft about testing, and they don't make their artifacts or platforms testable. There is a bit of diminishing returns to what I believe, but it also depends on ones dependencies and what one considers 100%. As an example, I designed a multi-TB file-format and synchronization engine. I made that engine a dependency free library such that all th…

Yes, I figure it would have been easy to get to 100% if MySQL had been designed so that plugins could be coverage tested.

For example, there might be an "environment" API which abstracts memory management and file I/O. The plugin could use this API instead of direct calls. Then if MySQL had an option to use a specific environment implementation for a plugin, then testers could drop in their own implementation under test.

Getting that sort of support by MySQL is an example of what I think you mean by "cultural challenge."

Re: The Myth of Code Coverage

#97

Post by theshrike79 »

xienze wrote: post #29
rzimmerman wrote: post #6

Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…

> Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Absolutely correct. But at the same time, Strive for full coverage that hits every line AND tests the appropriate scenarios and edge cases.

I'm pretty sure all those IDE generated getters and setters will work flawlessly even if I don't add unit tests for them.

They can stay red until the end of time for all I care.

Integration tests for the most common paths, unit tests for more complex logic.

Re: The Myth of Code Coverage

#98

Post by tcbasche »

jrockway wrote: post #55
Tomis02 wrote: post #41

Earlier quoted context omitted.

Really? You want to strive to test that logging logs and observability observes? You want to test constructors, getters and setters? Testing trivial code brings negative value, why would you do that?

I prefer to test everything. The parts you don't test are the parts that break. For example, here's a case where I incremented the wrong metric: https://github.com/jrockway/alertmanager-status/commit/fccae... I noticed the bug when I went to look at a dashboard with that metric on it, and noticed it had the wrong name. If go's Prometheus library had an easy way to run "metric.CurrentValue()", I would have tested it..…

Arguably you should be testing behaviour rather than getters and setters. If a codepath requires a name formatted correctly, then that forms the basis of a test and an assertion. Otherwise you're introducing fragility by coupling your 'under the hood' stuff with your 'API' (so to speak).

Re: The Myth of Code Coverage

#99

Post by tcbasche »

nuclear_eclipse wrote: post #84
dahfizz wrote: post #11

There is a big difference here between strongly typed, compiled languages and weakly typed interpreted languages. High code coverage is much more important on e.g. Javascript or Python because running the code is the only way you know that it "compiles" and you didn't do something dumb like mistake the type of arguments, access fields that don't exist, etc. In a compiled codebase, I think functional / end-to-end test…

This is where I've become a staunch proponent of type hinting Python code. I've caught more subtle bugs by enabling mypy on new sections of code than by running and debugging in production. Of course, this requires adding type hints everywhere, which can a struggle for older codebases, but MonkeyType is a great way to solve that, along with diligence on adding type hints to all new code you write, giving progressivel…

mypy is great but I've become enamoured with pyright[1] of late. It's super fast and catches way more problems than mypy does. It also catches syntax and semantic errors as a nice aside.

[1] https://github.com/microsoft/pyright

Post reply on HN