Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The tragedy of 100% code coverage (2016)

labs.ig.com

Post reply on HN
Print view View original

341–346 of 346 posts

Re: The tragedy of 100% code coverage (2016)

#341

Post by thehardsphere »

sqeaky wrote: post #266
thehardsphere wrote: post #152

Earlier quoted context omitted.

If we're being pedantic, there's a second common definition of the word tragedy that does not refer to a drama or literary work, but "an event causing great suffering, destruction, and distress, such as a serious accident, crime, or natural catastrophe." I think when you take all of the time wasted on useless tests written merely for the sake of having tests, that waste is tragic. You could be doing anything else wit…

I am curious if there is any good way to measure that amount of waste. I agree that waste from extra and useless tests exists, but I am not convinced that waste is entirely bad. It seems to me that waste is unavoidable with teams newer to automated testing and may simply be part of the cost of using automated tests. If that is the case then it seems better to compare the cost of bugs with no automated to cost of supe…

Well, if the organization tracks the time developers spend doing things, then it should be easy to estimate the waste.

1. Take the number of hours spent writing tests.

2. Multiply by whatever percentage of the tests are unnecessary.

3. Multiply by the labor cost per hour. Or revenue that was not made (e.g. if you could have billed those hours to a customer, but didn't).

The resulting number could either be a big deal or not, depending on how big your organization is and how much time you spent on superfluous tests.

Re: The tragedy of 100% code coverage (2016)

#342

Post by mannykannot »

mcv wrote: post #337
mannykannot wrote: post #285

Earlier quoted context omitted.

I was thinking that, for example, that a >= test is as valid as == in some cases. If == is actually used and is correct, substituting >= would not cause any valid test to fail. As I wrote this, however, I realized that if you substitute the inverse operator (!= for ==, = etc.) and it is not caught, you can infer it is not covered. (edit: or that, in the context of the program as a whole, the result of the operation i…

Substituting == for >= should cause a test to fail. Either == is correct, or >=. They can't both be correct. Should the two things always be equal? Then they shouldn't be unequal. Is one allowed to be bigger than the other? Then that should be allowed, and not rejected. If this change doesn't cause a test to fail, you're not testing edge cases for this comparison. (Of course that's assuming you want perfect coverage.…

Counter-example:

  while ( j = FINAL) break;
     ...
If, in the specific circumstances of this use, callback(j) will always equal FINAL before it exceeds it, a test for equality here will not cause the program to behave differently.

The fallacy of your argument is that == is not an assertion that its arguments should always be equal; it is a test of whether they are at some specific point in the algorithm.

Re: The tragedy of 100% code coverage (2016)

#343

Post by seabornleecn »

I think pursuing 100% test coverage is not a fixed state, it is a must have process to learn how to write tests.

Think about one question first: why did the manager force develop to achieve 100% coverage? There must have some benefits, or the manager might come from the competitor. When standing at a higher position, think of time and organization factors, it might be a good choice. If every engineer in the corporate has the deeply understanding of test coverage as the author, they really do not need to pursue 100% coverage. But in reality, we can see many companies which do not pursue test coverage, their coverage tend to be 0. That's why we need force 100% test coverage in a short time. Engineers need time to form the habit of test their code, and then experience the pain of bad tests. Then they start to think what kind of tests are valuable.

Re: The tragedy of 100% code coverage (2016)

#344

Post by ioquatix »

quantumhobbit wrote: post #6
ioquatix wrote: post #3

I have 100% code coverage on a couple of projects. It has two benefits: Behaviour is completely covered by tests, so changes in APIs which might break consumers of the library will at least be detected. New work on the library tends to follow the 100% coverage by convention, so it's somewhat easier to maintain. Apps that have 90% coverage, for example, tend to slip and slide around. Having 100% coverage projects the…

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…

Nope, if you submit a PR with crappy unit tests, I'll throw it politely back in your face and tell you to rewrite them :D

Re: The tragedy of 100% code coverage (2016)

#345

Post by ioquatix »

enraged_camel wrote: post #10
quantumhobbit wrote: post #6

Earlier quoted context omitted.

The problem is that coverage tools report whether that line of code executed and not whether its logic is correct. This can easily give you a false sense of security. So if I want to contribute to your project all I have to do is write some pointless tests that are sure to execute every single getter and setter method.(yes I have seen tests that exist solely to execute getters and setters). I don't have to actually t…

Exactly. It's like testing a newly built bridge by crossing it a hundred times with a bike. Sure, you "covered" it, but you sure as hell did not test it.

If you write unit tests like this, it's your own fault when the bridge collapses from driving a car over it.

Re: The tragedy of 100% code coverage (2016)

#346

Post by kelnos »

defined wrote: post #67
kelnos wrote: post #29

I find that, as I'm building something from scratch, the vast majority of the errors I make are just things I didn't think of. Tests don't help there because I can't test on input that I don't even imagine happening. So I generally write few tests, because, to be honest, most code is trivial and algorithm-light. Sure, if I have to write a parser or something a bit more fiddly, I'll write a unit test to be sure that i…

> An objection I hear to this is that you're not just writing tests for yourself, you're writing tests for the others who will need to help maintain your code, perhaps after you're gone. I'm somewhat sympathetic to this, but I would also say that if someone else needs to modify my code, they damn well better first understand it well enough such that they could write tests before changing it (if they deem it necessary…

Yeah, I don't disagree with that, and one of the things I do before I hand off a code base to others is beef up test coverage somewhat, especially in places where the expected behavior of a bit of code might need to be codified to a certain extent.

Regarding your "legacy code" quotes: I don't write code that isn't easy to change, or easy to read. If I write code that's hard to read, I delete it and start again. If I absolutely cannot write something that's easy to read, I write tests around it and document the hell out of it (I'm iffy on comments and docs as well, because they _always_ end up out of date, and then are more of a hindrance than no comments at all).

I think I'm pretty good at what I do, but I wouldn't consider myself a "god of programming". Writing clear, concise code isn't hard. It really really really isn't. In my experience, the main blocker to that is ridiculous time lines and pressure to ship. I know it can be hard to push back against that pressure, but you owe it to yourself, future maintainers of your code, and the company you work for (even if the company doesn't realize or appreciate it at the time) to slow things down and do things the right way. A former colleague used to say, "It's not right because it works; it works because it's right". Just because a bit of code produces the output you want, it doesn't mean it's right. Write the right code -- readable, maintainable, verifiable, testable -- and you don't even need to worry about it working, because of course it will.

I lean on type systems heavily. If I were writing python or ruby, I'd have 10x as much test code as application code, because I just do not believe you can trust a dynamic/weakly-typed language without them. This is why I avoid such languages; I think any gains in rapid development that you get from such languages are quickly wiped out by the need to write extensive tests, or, lacking those, all the bugs that come up because you don't have them.

These days I write most things in scala, if I can. No, it's not a perfect language (honestly, I'd say a half to two thirds of it is crap, but the rest of it is amazing), but it has a strong type system that lets you lean on the compiler so much more than many other languages. Simply the fact that it compiles gives me much higher confidence than with most other languages.

Post reply on HN