Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Myth of Code Coverage

preslav.me

Post reply on HN
Print view View original

51–60 of 115 posts

Re: The Myth of Code Coverage

#51

Post by breck »

dnautics wrote: post #49
breck wrote: post #46

Earlier quoted context omitted.

> I have to replace every time (it's snapshot-testing code-generating code), Do you store your snapshots separately? One note for devs reading this: I think snapshot tests are very cool, but I recommend trying them as a very separate thing than unit tests (if you are not already doing that). I removed all snapshot tests from our unit testing pipeline and it's much better for it. Was very surprised that Jest doesn't h…

Normally I would agree for webapps but in this case, it's for generating low-level code. In general, It's only a mistake if it's burdensome. I only have about 30 of these tests, and they are for critical code (I want the end user to be able to review the test and be confident that the generated code doesn't contain memory leaks, etc). https://github.com/ityonemo/zigler/blob/3de4d6fe4def265b689b...

Taking a quick peak at the code, maybe more smaller functions with fewer params to make it more testable?

Sorry that could be way off, just have limited time right now.

Re: The Myth of Code Coverage

#52

Post by xienze »

Tomis02 wrote: post #41
xienze wrote: post #29

Earlier quoted context omitted.

> Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Absolutely correct. But at the same time, Strive for full coverage that hits every line AND tests the appropriate scenarios and edge cases.

Really? You want to strive to test that logging logs and observability observes? You want to test constructors, getters and setters? Testing trivial code brings negative value, why would you do that?

> You want to strive to test that logging logs and observability observes?

Sure, why not? All that requires is running tests with the log level turned up. And it’s not unheard of that e.g. somewhere in a log statement someone forgot to check that the object they’re calling toString() on is not null.

Re: The Myth of Code Coverage

#53

Post by dnautics »

breck wrote: post #51
dnautics wrote: post #49

Earlier quoted context omitted.

Normally I would agree for webapps but in this case, it's for generating low-level code. In general, It's only a mistake if it's burdensome. I only have about 30 of these tests, and they are for critical code (I want the end user to be able to review the test and be confident that the generated code doesn't contain memory leaks, etc). https://github.com/ityonemo/zigler/blob/3de4d6fe4def265b689b...

Taking a quick peak at the code, maybe more smaller functions with fewer params to make it more testable? Sorry that could be way off, just have limited time right now.

I also do that, but since it's a LL programming language, there is coupling (a resource created in one place may be destroyed by different function).

Re: The Myth of Code Coverage

#54

Post by dnautics »

breck wrote: post #50
dnautics wrote: post #49

Earlier quoted context omitted.

Normally I would agree for webapps but in this case, it's for generating low-level code. In general, It's only a mistake if it's burdensome. I only have about 30 of these tests, and they are for critical code (I want the end user to be able to review the test and be confident that the generated code doesn't contain memory leaks, etc). https://github.com/ityonemo/zigler/blob/3de4d6fe4def265b689b...

But can't you identify the signal of the input and output and write small test cases to test just the signal? Snaps test the signal + noise. They help alert you when either changed, which is helpful if you think you may be missing tests for the signal, but ideally you would just make sure all the signal is tested, and the snaps would be just a later, independent sanity check tool. Obviously you know the details and o…

> They help alert you when either changed

that's exactly what I'm looking for =D. See response to sibling comment for why I also want to be able to review the full code layout.

Re: The Myth of Code Coverage

#55

Post by jrockway »

Tomis02 wrote: post #41
xienze wrote: post #29

Earlier quoted context omitted.

> Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Absolutely correct. But at the same time, Strive for full coverage that hits every line AND tests the appropriate scenarios and edge cases.

Really? You want to strive to test that logging logs and observability observes? You want to test constructors, getters and setters? Testing trivial code brings negative value, why would you do that?

I prefer to test everything. The parts you don't test are the parts that break.

For example, here's a case where I incremented the wrong metric:

https://github.com/jrockway/alertmanager-status/commit/fccae...

I noticed the bug when I went to look at a dashboard with that metric on it, and noticed it had the wrong name. If go's Prometheus library had an easy way to run "metric.CurrentValue()", I would have tested it... but it didn't, so I didn't. And then had to patch it, release it, and update all my servers. Writing the test (with that API) would have taken less than a second. Finding the bug in production and fixing it took an hour.

(That codebase has 100% line coverage, but of course, I know that 100% test coverage is a meaningless metric. That the code ran is a prerequisite to the program behaving correctly, but it also has to do the right thing.)

Re: The Myth of Code Coverage

#56

Post by mathgladiator »

postalrat wrote: post #27
mathgladiator wrote: post #9

I'm a strident adherent to having 100% code coverage, but this mostly works because my history in infrastructure and a desire to sleep well. The key idea that I have found that if you can't synthetically get your process into a specific state, then life is going to be hard. The problem with a lot of people is that they view testing as a burden rather than a criticism of the code they are testing. If you can't test yo…

You must be trolling.

I'm not. One of my side projects for example is a programming language where I have 100% code coverage.

From a career perspective, I have seeded new infrastructure services with 100% code coverage. At core, I believe achieving great reliability requires a solid foundation.

I don't claim these services are perfect, but when a bug is discovered then I can usually use the logs to figure out what state the program is in and then test the bug with a unit test. Then, I can protect future engineers from that issue.

Now, don't get me wrong, there are LOADS of silly tests for languages like Java to make code coverage tools happy. Like "new ClassThatOnlyHasStaticMethodsInIt()", but the key is that you can alarm on code coverage less than 100% but once you let the paper cuts build up it is hard to manage.

I'm a big believer in "slow is smooth, and smooth is fast" when it comes to building services for others.

Re: The Myth of Code Coverage

#57

Post by mathgladiator »

dalke wrote: post #43
mathgladiator wrote: post #9

I'm a strident adherent to having 100% code coverage, but this mostly works because my history in infrastructure and a desire to sleep well. The key idea that I have found that if you can't synthetically get your process into a specific state, then life is going to be hard. The problem with a lot of people is that they view testing as a burden rather than a criticism of the code they are testing. If you can't test yo…

I wrote a UDF for MySQL. It had just shy of 100% coverage. It used a malloc. I could not figure out how to trigger a malloc failure inside of a imported library running on a MySQL instance. Do I not check for malloc failure (in order to get 100%)? Or do I develop some sort of instrumented version of MySQL which lets me test malloc failure? That seemed far more overkill than warranted for the small project I worked on…

A core cultural challenge that we have to deal with is that people are very soft about testing, and they don't make their artifacts or platforms testable.

There is a bit of diminishing returns to what I believe, but it also depends on ones dependencies and what one considers 100%.

As an example, I designed a multi-TB file-format and synchronization engine. I made that engine a dependency free library such that all the math, contracts, threading, and scheduling could be 100% tested before integration. This forced the messy integration into the core application which then was E2E tested with hundreds of randomized tests.

The key concept at play was beating all the math into the ground such that it was damn near perfect because a single flaw would lose data.

Re: The Myth of Code Coverage

#58

Post by Supermancho »

Tomis02 wrote: post #41
xienze wrote: post #29

Earlier quoted context omitted.

> Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Absolutely correct. But at the same time, Strive for full coverage that hits every line AND tests the appropriate scenarios and edge cases.

Really? You want to strive to test that logging logs and observability observes? You want to test constructors, getters and setters? Testing trivial code brings negative value, why would you do that?

> You want to strive to test that logging logs and observability observes?

A test that calls code (which counts as code coverage) does not have to verify that something specific happened. Often, this is a weakness of a test. The test does not verify what you think it would for various reasons...visibility, access to resources, etc.

Re: The Myth of Code Coverage

#59

Post by trinovantes »

Back in my undergrad software testing course, we were taught test coverage in the context of a control flow graph.

I think most coverage measurements (at least in the GitHub projects where I've seen a coverage badge) only refer to statement coverage where it checks if every node of the CFG is reached. It's one of the weakest forms of testing and will rarely reveal the more obscure bugs that edge/(simple, prime, complete) path coverage testing can reveal.

I personally think the industry should stop using (node) coverage as some kind of golden metric.

Re: The Myth of Code Coverage

#60

Post by mannykannot »

hyperman1 wrote: post #7
ardy42 wrote: post #5

Earlier quoted context omitted.

> I was also going to say around 60-70 pct, but for a different reason: what's left in my code is mostly checking of assertions, debug logging and handling rare errors, i.e code that's not supposed to run. I grant you assertion-checking, but code for handling rare errors is not the code you should skip writing unit tests for. If it runs infrequently then you're far less likely to stumble on a regression during other…

Depends on rare errors. I meanthings like 'My database crashed halfway a transaction, my file system drops from under my application, my back end service gave up. You can't do much here. Dump some info, abrt the half- done work, maybe try again somewhere in the future. And yes, that last part migh deserve a test.

> ...abort the half-done work...

I have seen more than a few problems be caused by this not being done right. IIRC it was an element in the events that destroyed Knight Capital in just a few minutes.

If there is something in your code that can put your system in an inconsistent state, and your testing has not covered that scenario, then it is irrelevant what percentage of the code was covered by that testing. This combinatorial complexity and history-dependence is the reason why code coverage is a misleading indicator of quality: 100% code coverage is very far from 100% scenario coverage.

Post reply on HN