Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

191–200 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#191

/me raises hand on the pro-testing side I've been programming for a living since 1996, and only recently started to do TDD in the normal sense of writing unit tests before writing code. I've found to it to be an enormous help with keeping my code simple - the tests or the mocking getting difficult is a great indicator that my code can be simplified or generalised somehow I argued for functional instead of unit testin…

+1 - speaking as an ex-sceptic, the only people I know that don't like unit testing and/or TDD are the people who either misunderstand their purpose or have not learned how to utilise them properly yet.

Once TDD "clicked," the quality of my code shot up very rapidly indeed.

Re: The tragedy of 100% code coverage (2016)

#192
post #45

My main issue with unit testing is what defines a unit? Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression. Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box. That…

I agree with you. That's called functional testing, and it is very useful, but it is not unit testing. Unit testing: Test all methods and paths of a class, even private ones. Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests. ROI of functional testing is high, as it is usually done with real data. In m…

I think the problem lies with the definition of "unit" which you quote above. As with designs stacking on top of one another at various levels, one level's functional test is often the next level's unit test. Your app's unit test might mock the very same things that are covered by the functional test for an underlying library. That library's unit tests might mock the very same things that are covered by the functional tests for one of its own dependencies, including the OS. And so on, even down to the level of functional units within a chip. (Chip verifiers have a lot to teach us software types about this kind of thing BTW, and I know because I've worked with a few.)

The distinction I always try to make is whether you're testing a contract or an implementation. If you're testing a contract, ROI is likely to remain high even as you move to finder granularities. If you're testing an implementation, so that the tests fail even when the implementation is 100% correct, then ROI falls off a cliff.

That's exactly what happens when the "unit" gets too small. If certain private methods can't be called a certain way because of constraints imposed by the class's public methods, then testing those calls is testing things that can't happen. That takes time away from testing (and implementing fixes for) things that can happen. If the class's contract changes so that those internal methods can be called in new ways, then yes, a 100% code-coverage unit test might catch the error. So, in all likelihood, would the new functional test accompanying the public-method change. The delta between the two, times the likelihood of such a scenario occurring in the first place, is too small to justify the cost of writing those tests and the likelihood that they'll generate false negatives.

Re: The tragedy of 100% code coverage (2016)

#193

Earlier quoted context omitted.

This hits home with me. Incompetent developers need dogmatisms (100% unit coverage!!!! etc)

No, inexperienced developers need dogmatisms. Subtle difference perhaps, but it's worth remembering that every competent developer started out being inexperienced, and that it's nothing to be ashamed of.

No. We fetishize experience too much. I've known a great many competent inexperienced developers who could weigh tradeoffs and use the right approach to a given task, and a far too many highly experienced developers who destroyed productivity through dogmatism.

Re: The tragedy of 100% code coverage (2016)

#194
post #176

Earlier quoted context omitted.

How do you test it by brute force? How is the algorithm supposed to know what the right thing to return is, unless you rewrite the whole thing, correctly this time? And, if you've done that, just replace the actual function with the test and be done with it.

Your comparison source could be running on a different platform, or be slower, or ... and thus not be a drop-in replacement. (E.g. in the example of a broken floating-point unit, compare the results with a software emulation)

Hmm, true, it may have been a different platform/slower, thanks. It can't have been hardware/software, as the GP said they were testing a library (which implies software).

Re: The tragedy of 100% code coverage (2016)

#195

Earlier quoted context omitted.

If we're being pedantic, a tragedy is _a drama or literary work in which the main character is brought to ruin or suffers extreme sorrow, especially as a consequence of a tragic flaw, moral weakness, or inability to cope with unfavorable circumstances_. Definitely not a tragedy.

If we're being pedantic, there's a second common definition of the word tragedy that does not refer to a drama or literary work, but "an event causing great suffering, destruction, and distress, such as a serious accident, crime, or natural catastrophe." I think when you take all of the time wasted on useless tests written merely for the sake of having tests, that waste is tragic. You could be doing anything else wit…

I agree, I was just kicking GP's pedantry up a notch.

Re: The tragedy of 100% code coverage (2016)

#196
post #189
post #183

Earlier quoted context omitted.

No, because I have worked with several dozen code bases, hundreds if not thousands of bugs, and Unit Tests are if anything a bad sign associated with code in desperate need of refactoring. They are the uncanny valley of quality. Ed: Unit Tests become more valuable as code quality decreases.

Hmm... You're capitalizing "Unit Tests" like there's a need to distinguish the difference between those and what I infer is "code that was not Unit Tests that we ran to verify correctness". Am I correct? Are you drawing a distinction? Or did you truly have no code used to verify correctness? Or no automated way to test?

There are many types of tests, Integration tests are different than Unit Tests. Having a script click on UI elements is fundamentally different from isolating a function to verify it does what it should.

Re: The tragedy of 100% code coverage (2016)

#197

The worse the developer, the more tests he'll write. Instead of writing clean code that makes sense and is easy to reason about, he will write long-winded, poorly abstracted, weird code that is prone to breaking without an extensive "test suite" to hold the madness together and god forbid raise an alert when some unexpected file over here breaks a function over there. Tests will be poorly written, pointless, and give…

> The worse the developer, the more tests he'll write. As always, generalization is the tool of the fool (sorry for the fool part, but it rhymes ;) ). Writing pointless stubs / mocks and testing execution order of statements is definitely a bad pattern, writing many and good functional, e2e and integration tests however is not.

Generalization is at the heart of science. The lack of generalization is one of the most frustrating attacks you can launch on a scientist's empiricism.

Re: The tragedy of 100% code coverage (2016)

#198
post #69

Earlier quoted context omitted.

I'm curious - how old are the systems you're working on? In my experience, unit tests don't catch many bugs when the code is fresh. But when it's five years old with many modifications over the code base, some dumb little test that you thought was a waste of time is now alerting you to what would have been a horror regression. In other words: Even though it feels like it's slowing you down now, if you write tests whi…

I work in the public sector in Scandinavia and some of our oldest systems still in service run on an old tandem. So some of it is pretty old. This gives us some unique abilities in terms of modeling our productivity of course, because we started measuring before anyone thought up unit testing. Over the past 15 years, unit testing has failed to produce anything positive, and test driven development has been an absolut…

Do you have any metrics that you can share?

Re: The tragedy of 100% code coverage (2016)

#199
post #45

My main issue with unit testing is what defines a unit? Throughout my career I find tests that tests the very lowest implementation detail, like private helper methods, and even though a project can achieve 100% coverage it still is no help avoiding bugs or regression. Given a micro service architecture I now advocate treating each service as a black box and focus on writing tests for the boundaries of that box. That…

I agree with you. That's called functional testing, and it is very useful, but it is not unit testing. Unit testing: Test all methods and paths of a class, even private ones. Functional Testing: Test the public api of a class/service only. If something is wrong internally, it will be caught without having to write countless of little tests. ROI of functional testing is high, as it is usually done with real data. In m…

>Unit testing: Test all methods and paths of a class, even private ones.

I disagree with this. Unit testing means testing the "unit" (i.e. the class/object) but still at a boundary level: the public API of that class. You should mock out any dependencies that object would have to ensure you are only testing that class but the class API has to be respected.

A private method is a hidden implementation detail, testing those would be overfitting your tests to the current implementation meaning if you change one character anywhere in your source you will almost certainly have to change one or more tests. Plus, if your tests are so tightly coupled to the implementation, it's likely to suffer from any bugs the implementation does causing them to be hidden (never forget that tests are also code and therefor have bugs at a similar rate to any other code).

Writing test code requires the same level (if not more) of engineering discipline that writing the code does.

>ROI of functional testing is high, as it is usually done with real data.

I also disagree with all of this. Functional testing is a kind of sanity check that the parts actually work together once assembled. If you have proper unit test coverage (and properly designed/engineered tests!) the functional testing is basically checking configuration. The problems with function testing are (1) testing is about checking code paths but which paths functional tests take can be hard to predict and differ between subsequent runs, making it hard to make any statement about what passing actually means. (2) is exactly what you mentioned as a positive: people tend to want to use what they call "real" data, i.e. data they have actually seen before. Which means it's probably only good for catching bugs they know about, not ones they've never seen before.

>In my opinion unit testing is a huge waste of time.

I would actually agree to this because I think answering the question of "how do we detect bad code" with "write more code" is problematic. I'd rather go the Haskell/Idris route and be able to prove that I have no bugs.

>Most of the tests devolve int mock objects, calling mock methods, and doing this that really don't help to find real world bugs, where two unit tests pass, but their methods produce the wrong output.

This kind of response sounds like a self fulling prophecy. The exact point of mocking is to test code paths, i.e. if this dependency returns this result how will the class under inspection behave in response. Ideally you would use mocks to test everything that every dependency could respond. Unless the types involved have very few inhabitants, this isn't generally possible (even programatically) but the closer you get to this the more you can trust your tests.

Re: The tragedy of 100% code coverage (2016)

#200

Earlier quoted context omitted.

So you're saying test coverage is a negative? It's not an either or.

If I am understanding correctly, he is saying that well-written code requires fewer tests in order to be fully tested. Fully tested is better than untested, assuming that code is fully tested, well written code will require fewer tests. That is my understanding, as it is rather hard to extract a solid argument from his posts.

He specifies "any tests", as do several other anti-test posts in this discussion, though often someone will reply to them and assume, like you have done, that they have another complex set of automated tests that they're happy about, and they're only railing against some unspecified subset of automated tests. I'm not sure if that's being over-charitable or not.
Post reply on HN