Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Myth of Code Coverage

preslav.me

Post reply on HN
Print view View original

81–90 of 115 posts

Re: The Myth of Code Coverage

#81

Post by hinkley »

maerF0x0 wrote: post #64

> Being dogmatic about tests and covering every line will only make it more difficult to get rid of it. This is only true if one is lackadaisical about how they architect their code and (unit) tests. You should be able to completely smash a function and it's tests without breaking any other tests ... If not then you're testing a different unit within those broken unittests.

Someone on my team likes to write their own mocks instead of using the mocking framework. A lot of people don’t really understand the point of testing, but this behavior is pretty far down on the spectrum.

The mock is hard to write by hand, so they get sunk cost fallacy and share it between tests. Now the tests are coupled to each other, which makes it hard to change features. But wait, there’s more.

They made space for some delegation/composition that never arrived or disappeared, and so they’re sharing the same mock across two different test suites, in two different directories. And made very coarse grained commits, so even if I wanted to know how tf they got here, which I really don’t, I’d have trouble doing it.

Writing complex scaffolding for your tests is supposed to hurt. Pain is your body telling you something is wrong. If you can’t set the preconditions in a few lines then you don’t understand the problem, or you don’t understand your own code. Both are bad for your coworkers.

Big decisions are the culmination of a bunch of little decisions. They are not the little decisions themselves. Don’t write your code like they are, and all of this shit gets ten times easier.

Re: The Myth of Code Coverage

#82

Post by hinkley »

dnautics wrote: post #44
breck wrote: post #34

> Being dogmatic about tests and covering every line will only make it more difficult to get rid of it. I find the opposite to be the case. I'm very comfortable tossing away a bunch of code with great test coverage. You can always cherry-pick it back later and know that it works. (I'm not 100% purist, but certainly I'd say 80-90+% and not 66%). The important thing is you should be so fast at writing testable code tha…

> You can always cherry-pick it back later and know that it works. Exactly, "this is what git is for". I think being uncomfortable tossing tested code is more of a reflection of the temperament of the person saying it than any truism about coders in general. If anything a little bit of a roadblock with test cases can be a good thing. "By deleting this test I am really certain I want to remove this feature". There's o…

I find that the way a test is written substantially affects people’s willingness to delete them, so while there is some truth to what you say, it’s not the entire picture.

Near as I can guess, it’s related to the anchoring. I’m replacing a few tests with a new set of tests. How complicated should they be? The old test was simple, I should replace it with a simple test. I need six of them, six simple tests ain’t so bad. Nearly took me longer to think about it than to do it.

If I write three line tests, the test quality tends to erode slowly. If I left sketchy tests, they turn to garbage disturbingly fast.

Re: The Myth of Code Coverage

#83

Post by hinkley »

npollock wrote: post #61

What types of solutions exist for automating the creation of front-end tests? Either full automation, or partial with some user input/direction. I found https://kwola.io/ but wondering if there are other alternatives

If you could automatically test a thing, what’s the line between generating tests and doing static analysis?

Re: The Myth of Code Coverage

#84

Post by nuclear_eclipse »

dahfizz wrote: post #11

There is a big difference here between strongly typed, compiled languages and weakly typed interpreted languages. High code coverage is much more important on e.g. Javascript or Python because running the code is the only way you know that it "compiles" and you didn't do something dumb like mistake the type of arguments, access fields that don't exist, etc. In a compiled codebase, I think functional / end-to-end test…

This is where I've become a staunch proponent of type hinting Python code. I've caught more subtle bugs by enabling mypy on new sections of code than by running and debugging in production. Of course, this requires adding type hints everywhere, which can a struggle for older codebases, but MonkeyType is a great way to solve that, along with diligence on adding type hints to all new code you write, giving progressively more benefits as more of the codebase is covered by type annotations.

Re: The Myth of Code Coverage

#85

Post by hinkley »

pmb wrote: post #42

100% is the way to be. Once you get there, then you can start having useful conversations about how to make the coverage more meaningful. But first, you have to make sure that every line is run at least once without crashing (except for the lines that are supposed to crash things - those you need to verify DO crash things). Interestingly, once something is designed for testability it is more likely to not have bugs.…

Disagree, empirically. Once you bury the needle, nobody wants to talk about whether the tests are garbage, because doing so means that you have to admit to misrepresenting things to management. So you’re either calling people out or complicit in the lie. People don’t want to look at these actions honestly, so they deflect to protect their egos.

None of these dynamics are an issue when you’re at 80% code coverage. It’s easier to fix how we do things while still slowly raising the stats.

You’re trying to run when you haven’t learned to walk (or with a few teams I’ve seen, crawl).

Re: The Myth of Code Coverage

#87

Post by zlynx »

hinkley wrote: post #77
rzimmerman wrote: post #6

Just because code is touched during the test process doesn’t necessarily mean it has been tested. Coverage is more useful for finding chunks of code that aren’t exercised at all by tests. Branches that never get hit that probably deserve extra scrutiny. Coverage is an interesting heuristic but 100% code coverage is not 100% bug free code. Likewise, there’s stuff that’s just not worth it to wrap in a test. The effort…

And that’s why you shouldn’t chase after code coverage percentages. While there are uncovered lines, you have a TODO list that has been generated by the machine, telling you what needs more attention. If you go to that section intending to test it well, you’ll get a lot of lines of test code for a modest increase in coverage. If you’re just trying to boost coverage, you’ll check off the todo without accomplishing muc…

Except when handling an error condition from a hard to test system call that was previously ignored.

Being forced into stupid tricks like adding five useless log statements so that the CI tool doesn't block your change is ... annoying.

Re: The Myth of Code Coverage

#88

Post by golergka »

Conversations like this a meaningless without first determining what kind of software are we talking about. If it's a first version made quickly to test an idea, the ideal would be 0%. Rapidly developing product, 40%, mature product, 60%, database (like Postgres or SQLite) - 100%, and any software that deals with money, health and airspace should have each live of code covered at least three times by completely different tests, both unit and integration.

There's not one size that fits all in software development. And if you don't specify what software are you talking about, everybody will explicitly talk about their own domain and experience.

Re: The Myth of Code Coverage

#89

Post by TeaVMFan »

whymarrh wrote: post #12
g14i wrote: post #8

Earlier quoted context omitted.

Totally agree. In one of our projects (around 200K Python LoC) we had 90% CodCov, but we frequently found some bugs. Recently we started to heavily use mutation testing and fuzzing to find edge cases on parts of the code that were already "covered" according to the CodCov report. It was definitely worth it. I highly recommend investing in mutation and fuzzy testing.

I was just about to bring up mutation testing. I've had some pretty great success with PIT [1] when writing Java. Code coverage and mutation test coverage pair wonderfully together. [1]:https://pitest.org

Mutation coverage definitely improves the testing process. It is much harder to accidentally write bad tests once you are using a tool like PIT: https://frequal.com/java/HighQualityTestsWithPitMutationTest...

Re: The Myth of Code Coverage

#90

Post by smithza »

Loosely coupled code could well have unit testing and mocking covering >90% with great results. We should not conflate unit and integrations tests here. I have also found regression testing to be a great way to declare and interpret intent when trying to understand someone else's code.

I think the classic book "Working Effectively with Legacy Code" declared "legacy code" to simply be code that has no tests.

Post reply on HN