Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Myth of Code Coverage

preslav.me

Post reply on HN
Print view View original

101–110 of 115 posts

Re: The Myth of Code Coverage

#101

Post by tcbasche »

bluGill wrote: post #39
dudul wrote: post #15

Earlier quoted context omitted.

That's interesting because I think the exact opposite. For dynamically typed languages I think it's more valuable to have end-to-end tests to make sure that the whole pipeline works correctly. I want to validate that all the call sites for function F are passing an int, as intended, as opposed to a string. For statically typed languages, I already know that all call sites for F are passing an int. So I want to unit t…

Dynamically types languages are just plane unsuitable for the type of large project where you need to break your end to tests down into unit tests. If you have a trivial program (under 100,000 lines of code or so), your end to end tests won't take that long to run if you take care, you will never be able to maintain that much in a dynamic language just because change becomes so hard without static types.

That's not really true, change is incredibly _easy_ without static types. I've worked an a few large projects now all in Python or JS where the test time has never really been an issue, nor has the ability to make changes.

I'd say it's not really the language, but rather the abuse of it. The same goes for any project in any programming language.

Re: The Myth of Code Coverage

#102

Post by UK-Al05 »

hinkley wrote: post #81
maerF0x0 wrote: post #64

> Being dogmatic about tests and covering every line will only make it more difficult to get rid of it. This is only true if one is lackadaisical about how they architect their code and (unit) tests. You should be able to completely smash a function and it's tests without breaking any other tests ... If not then you're testing a different unit within those broken unittests.

Someone on my team likes to write their own mocks instead of using the mocking framework. A lot of people don’t really understand the point of testing, but this behavior is pretty far down on the spectrum. The mock is hard to write by hand, so they get sunk cost fallacy and share it between tests. Now the tests are coupled to each other, which makes it hard to change features. But wait, there’s more. They made space…

The mock should display the same behaviour as the real thing.

So I don't think sharing one is bad.

The ideal fake is one that shares the same behaviour as the real thing. And you have tests that run across both to validate this.

Quite frankly mocks become the bane of large code bases.

I don't know how many times I've seen tests that are basically useless because the mock behaves nothing like how it behaves in the real world. Like throwing an exception for not found rather than null.

Keeping those contracts in sync are much easier with fakes.

Re: The Myth of Code Coverage

#103

Post by smichel17 »

BeetleB wrote: post #72
postalrat wrote: post #27

Earlier quoted context omitted.

You must be trolling.

I don't think so. I once took on the challenge of writing as many unit tests as possible for a project at work (the project did not have unit tests - but was well covered with other types of tests). The two key takeaways I got from the effort: First, I had no idea how coupled my code was until I tried writing many unit tests for it. If it's hard for you to instantiate your class without involving N other libraries/ob…

Could you give an example of the second part?

Re: The Myth of Code Coverage

#104

Post by GuB-42 »

> The only sure-fire way to improve code coverage (and by that keep software relevant) is to identify and remove the unnecessary code.

I remember what I think is a demoscene group taking that to the extreme. They ran coverage analysis and removed all uncovered code, thus achieving 100% coverage.

The point was to fit all their code in an very small executable (typically 4k or 64k). It was absolutely terrible for robustness: all edge cases are ignored. But since it was a demoscene production, it didn't matter, but if it was a server, that would have made it the most exploitable code ever.

Re: The Myth of Code Coverage

#105

Post by hinkley »

UK-Al05 wrote: post #102
hinkley wrote: post #81

Earlier quoted context omitted.

Someone on my team likes to write their own mocks instead of using the mocking framework. A lot of people don’t really understand the point of testing, but this behavior is pretty far down on the spectrum. The mock is hard to write by hand, so they get sunk cost fallacy and share it between tests. Now the tests are coupled to each other, which makes it hard to change features. But wait, there’s more. They made space…

The mock should display the same behaviour as the real thing. So I don't think sharing one is bad. The ideal fake is one that shares the same behaviour as the real thing. And you have tests that run across both to validate this. Quite frankly mocks become the bane of large code bases. I don't know how many times I've seen tests that are basically useless because the mock behaves nothing like how it behaves in the rea…

You can’t change or reorganize the behavior of the real thing without changing everything that using it all at once. This is challenging enough without elaborate handwritten mocks, but much worse with them. Most people don’t have the sort of stubbornness it takes to overcome this sort of adversity. So you’ve locked in your tech debt even harder.

If you have one function simultaneously using large parts of another chunk of code, such that you feel like you should write logic instead of stubs to simulate it, you already have a huge coupling problem that you should fix, instead of shoveling more code after bad. That’s what I mean by “it should hurt.” The friction is not a bug, it’s a feature. Slow your roll and look at your busted architecture, instead of cementing it in place with a layer of tests.

The other difficulty with mutating your mocks or otherwise changing testing tools is with negative tests. The changes can and sadly do end up creating tests that can’t fail (evergreen) but still increase coverage and confidence. Like a safety railing that has corroded, or a broken smoke detector. Simple mocks that exist entirely within the test, or the suite at farthest, are more amenable to change. Our job is change, when you get right down to it.

Re: The Myth of Code Coverage

#106

Post by BeetleB »

smichel17 wrote: post #103
BeetleB wrote: post #72

Earlier quoted context omitted.

I don't think so. I once took on the challenge of writing as many unit tests as possible for a project at work (the project did not have unit tests - but was well covered with other types of tests). The two key takeaways I got from the effort: First, I had no idea how coupled my code was until I tried writing many unit tests for it. If it's hard for you to instantiate your class without involving N other libraries/ob…

Could you give an example of the second part?

It's both project and language specific. A trivial example is C++ classes with private methods. Some people do a hack that when compiling for testing converts all private methods/attributes to public. This way it's easy to test private methods individually. Please don't do this.

Unit testing C++ is not as easy as in some other languages. The language is fairly rigid. Some people make heavy use of friend classes to assist with testing, but this can be overdone (indeed, many C++ developers are against any use of friend).

Re: The Myth of Code Coverage

#107

Post by postalrat »

mathgladiator wrote: post #56
postalrat wrote: post #27

Earlier quoted context omitted.

You must be trolling.

I'm not. One of my side projects for example is a programming language where I have 100% code coverage. From a career perspective, I have seeded new infrastructure services with 100% code coverage. At core, I believe achieving great reliability requires a solid foundation. I don't claim these services are perfect, but when a bug is discovered then I can usually use the logs to figure out what state the program is in…

What I often see is people want 100% coverage AND want tests to run fast. And often that leads to using mocks and other techniques. Once you go down that road you end up with brittle test and your developers spend a lot of time writing and repairing tests.

I'm not against testing but feel people waste a lot of time writing tests that are more a liability then an asset. I'm hoping we get better tools soon and people look back and wonder WTF were people thinking.

Re: The Myth of Code Coverage

#108

Post by maerF0x0 »

hinkley wrote: post #81
maerF0x0 wrote: post #64

> Being dogmatic about tests and covering every line will only make it more difficult to get rid of it. This is only true if one is lackadaisical about how they architect their code and (unit) tests. You should be able to completely smash a function and it's tests without breaking any other tests ... If not then you're testing a different unit within those broken unittests.

Someone on my team likes to write their own mocks instead of using the mocking framework. A lot of people don’t really understand the point of testing, but this behavior is pretty far down on the spectrum. The mock is hard to write by hand, so they get sunk cost fallacy and share it between tests. Now the tests are coupled to each other, which makes it hard to change features. But wait, there’s more. They made space…

> A lot of people don’t really understand the point of testing, but this behavior is pretty far down on the spectrum.

I did "unit testing" and dependency injection of "mocks" for years before I actually gave them the thought and learning they required. Frequently we treat testing as a barrier to the the already "good enough" code we want to put in production. We miss that the larger production is (feature wise), the worse the tradeoff of marginally adding one feature to potentially taking many/all down...

Reading the sinon.js documentation https://sinonjs.org/releases/v9.2.1/ really helped clarify the roles of various spies,stubs,mocks et al that IMO is required to

Re: The Myth of Code Coverage

#109

Post by postalrat »

breck wrote: post #37
postalrat wrote: post #27

Earlier quoted context omitted.

You must be trolling.

> " If you can't test your code easily, then your code is awful. End of story." I don't think this is trolling. Maybe I would have phrased it in a nicer way, but I agree with their math. Now I don't think you need to get to 100%, but if you can't easily see how you could get to 100%, that's a problem.

How easy is it test a function add(a, b) that just adds a and b and returns the result?

Or how bout testing:

  function divide(a, b, fn) {
    let [_a, _b] = fn(a, b);
    return _a / _b;
  }

Re: The Myth of Code Coverage

#110

Post by breck »

postalrat wrote: post #109
breck wrote: post #37

Earlier quoted context omitted.

> " If you can't test your code easily, then your code is awful. End of story." I don't think this is trolling. Maybe I would have phrased it in a nicer way, but I agree with their math. Now I don't think you need to get to 100%, but if you can't easily see how you could get to 100%, that's a problem.

How easy is it test a function add(a, b) that just adds a and b and returns the result? Or how bout testing: function divide(a, b, fn) { let [_a, _b] = fn(a, b); return _a / _b; }

I meant that's an awful function. I never, ever, ever have a single letter identifier. So I can't respond yet.
Post reply on HN