Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

11–20 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#11

Post by mjevans »

I think I'd rather focus on documenting the information flow. Of having the tools to track down where things start to go wrong when there's a problem and I ask things to run with more verbosity.

Initial "complete coverage" should probably start from mockups that test an entire API. The complete part should be that, in some way, the tests cover expected successes AND failures (successfully return failure) of every part of the API, but there's no need to test things individually if they've already been tested by other test cases.

Invariably reality will come up with more cases and someone will notice an area that wasn't quite fully tested. That's where a bug exists, but the golden test cases probably wouldn't have located it anyway. It'll take thousands or millions of users to hit that combination and notice it. Then you get to add another test case while you're fixing the problem.

Re: The tragedy of 100% code coverage (2016)

#12

Post by ben_jones »

quantumhobbit wrote: post #6
ioquatix wrote: post #3

I have 100% code coverage on a couple of projects. It has two benefits: Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected. New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the…

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…

A good example would be in dynamically typed languages where you can hit a code path but forget to validate a certain input type which could cause a bug.

Or one I've been personally hating lately, a nil pointer dereference in Golang from a well tested library that didn't do proper error propagation.

Re: The tragedy of 100% code coverage (2016)

#13

Post by kabdib »

I've seen projects where management had rules like "you must have 70% code coverage before you check in". Which is crazy, for a lot of reasons.

But the developer response in a couple cases was to puff the code up with layers of fluff that just added levels of abstraction that just passed stuff down to the next layer, unchanged, with a bunch of parameter checking at each new level. This had the effect of adding a bunch of code with no chance of failure, artificially increasing the amount of code covered by the tests (which, by the way, were bullshit).

I got to rip all that junk out. It ran faster, was easier to understand and maintain, and I made sure I never, ever worked with the people who wrote that stuff.

Re: The tragedy of 100% code coverage (2016)

#14

Post by ben_jones »

hibikir wrote: post #9

I might be completely wrong on this one, but it seems to me that a lot of the precepts of TDD and full code coverage have a lot to do with the tools that were used by some of the people that popularized this. Some of my day involves writing Ruby. I find using Ruby without 100% code coverage to be like handling a loaded gun: I can track many outages to things as silly as a typo in an error handling branch that went un…

IMO ruby is the worst because its so easy, and often times encouraged, to write layers of useless tests that ultimately give only a false sense of security. I recently re-did the Michael Hartl tutorial to catch up with Rails5 and it gives examples where you test proper template rendering. I love that book but I'd rather shoot myself in the foot then spend dev time writing tests to check if the right html title attribute was rendered for the page...

Re: The tragedy of 100% code coverage (2016)

#15

Post by coding123 »

I wish people cared more about the craft of an amazing plugin architecture or an advanced integration between a machine learning system and a UI, but no, more and more of our collective development departments care more about TDD and making sure things look perfect. Don't worry about the fact that there are no integration tests and we keep breaking larger systems, and while there might be 100% code coverage, no developer actually understands the overall system.

Re: The tragedy of 100% code coverage (2016)

#16

Post by penpapersw »

I think a bigger epidemic is we're putting too much emphasis on "do this" and "do that" and "if you don't do this then you're a terrible programmer". While that sometimes may be true, much more importantly is to have competent, properly trained professionals, who can reason and think critically about what they're doing, and who have a few years of experience doing this under their belt. Just like other skilled trades, there's a certain kind of knowledge that you can't just explain or distill into a set of rules, you have to just know it. And I see that in the first example in this article, where the junior programmer is writing terrible tests because he just doesn't know why they're bad tests (yet).

Re: The tragedy of 100% code coverage (2016)

#17

Post by penpapersw »

ben_jones wrote: post #14
hibikir wrote: post #9

I might be completely wrong on this one, but it seems to me that a lot of the precepts of TDD and full code coverage have a lot to do with the tools that were used by some of the people that popularized this. Some of my day involves writing Ruby. I find using Ruby without 100% code coverage to be like handling a loaded gun: I can track many outages to things as silly as a typo in an error handling branch that went un…

IMO ruby is the worst because its so easy, and often times encouraged, to write layers of useless tests that ultimately give only a false sense of security. I recently re-did the Michael Hartl tutorial to catch up with Rails5 and it gives examples where you test proper template rendering. I love that book but I'd rather shoot myself in the foot then spend dev time writing tests to check if the right html title attrib…

Sometimes the right test is just occasional manual testing. And sometimes it's even just the fact that no users have complained about something obvious not being completely wrong.

Re: The tragedy of 100% code coverage (2016)

#18

Post by cbanek »

I've had to work on mission critical projects with 100% code coverage (or people striving for it). The real tragedy isn't mentioned though - even if you do all the work, and cover every line in a test, unless you cover 100% of your underlying dependencies, and cover all your inputs, you're still not covering all the cases.

Just because you ran a function or ran a line doesn't mean it will work for the range of inputs you are allowing. If your function that you are running coverage on calls into the OS or a dependency, you also have to be ready for whatever that might return.

Therefore you can't tell if your code is right just by having run it. Worse, you might be lulled into a false sense of security by saying it works because that line is "covered by testing".

The real answer is to be smart, pick the right kind of testing at the right level to get the most bang for your buck. Unit test your complex logic. Stress test your locking, threading, perf, and io. Integration test your services.

Re: The tragedy of 100% code coverage (2016)

#19

Post by dcw303 »

I think you should write a test.

Naming the test just "initialise" is not very useful as it doesn't assert what you expect the method under test to do. Given that the purpose of the initialise function is to populate a watchlists collection variable from the parameter, i'd name the test something like "initialise_daoRecordCountIs9_watchlistCountIs9". The pattern I generally use is __.

Then, my test would be the following:

* Set up / mock the dao parameter to have 9 rows

* Create an instance of the class under test and push in the dao parameter

* Verify / Assert that the class under test now has 9 items in the watchlists variable - I'm assuming there is a public method to access that.

Post reply on HN