The author is correct, in the sense that over the lifetime of a software project 90% of the value from unit testing will come from 10% of the tests. The problem is that you won't know in advance which 10% that's going to be, so you have to write and maintain all of them.
Why Most Unit Testing Is Waste [pdf]
131–140 of 159 posts
Re: Why Most Unit Testing Is Waste [pdf]
#132Earlier quoted context omitted.
This. We tend to put a high value on code review, yet some don't value unit testing. Lean in and I'll tell you a secret: ... unit testing is automated, repeatable, code review!
I don't see such a large overlap between code review and unit testing. Here's an example from one of my code reviews, where I discovered two issues: 1) A relaxed requirement was agreed to by the PO, but the remote developer wasn't aware of that and wrote code to fulfil the original complex requirement. The implementation was harder to understand and touched more areas of the code, leading to issue number 2. 2) By ana…
Static analysis is another tool in the toolbox, and invaluable as well.
Re: Why Most Unit Testing Is Waste [pdf]
#133Earlier quoted context omitted.
> Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Let's assume you hire good programmers, because otherwise you're doomed. But oftentimes, the "good" programmer and the "bad" programmer are the same person, six months apart: 1. John writes some good code, with good integration tests and good unit tests…
Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Were the failing tests due to John/Jane's correctly coded changes, regressions, or bad code changes? The tests provide no meaningful insight into that - it's still ultimately up to the programmer to make that value judgement based on the understanding of what the…
>> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed".
Even if all you know is "something changed", that's valuable. Pre-existing unit tests can give you confidence that you understand the change you made. You may find an unexpected failure that alerts you to an interaction you didn't consider. Or maybe you'll see a pass where you expected failure, and have a mystery to solve. Or you may see that the tests are failing exactly how you expected they would. At least you have more information than "well, it compiled".
And if "the unit tests are providing no actual indication that their changes were the correct changes to make", even the staunchest proponents of TDD would probably advise you to delete those tests. If there's 1 thing most proponents and opponents of unit testing can agree on, it's probably that low-value tests are worse than none at all.
I have no idea if this describes you, but I've noticed a really unfortunate trend where experienced engineers decide to try out TDD, write low-value tests that become a drag on the project, and assume that their output is representative of the practice in general before they've put the time in to make it over the learning curve. People like to assume that skilled devs just inherently know how to write good unit tests, but testing is itself a skill that must be specifically cultivated.
Re: Why Most Unit Testing Is Waste [pdf]
#134Earlier quoted context omitted.
The article addresses your first point in a big section: 1.4 The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process Your other point about only good programmers not needing unit tests is moot as you haven't followed it through to the conclusion. Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. S…
My point was that even bad unit tests are better than both good/bad word docs or no unit tests. So even bad tests written by bad programmers have a value. They have a large COST too (which is what he's arguing), my argument was merely that the value might not be less than the cost, which is his assertion. > Software engineering research has shown that the most costeffective > places to remove bugs are during the tran…
Any idiot can ship version 1.0. I get exasperated with people who try to act like they have some special wisdom to share about getting the first version out. Yes, there are some specific things you need to do that some of us lack, but you can find those in very nearly every single self-help or motivational book.
It takes some proper skill, vision, and maturity to ship version 3.0. And you probably won’t get a big payday if you can’t.
Re: Why Most Unit Testing Is Waste [pdf]
#135I'm afraid that the TDD movement went to far. The proponents became so convinced that they were right that they started comparing themselves to people who argued that washing hands was important in early surgery, and that those who questioned it would soon be unemployable in the field. Not those who didn't practice it, those who questioned it. I dealt with some of this, including a rather bullying type who tried to b…
As in many areas of thought, when the pendulum swings too far in one direction, it then swings too far in the other direction. Hopefully we'll arrive at a reasonable middle ground at some point, and unit tests will be valued (and prioritized) neither too much nor too little. On a side note, unfortunately I think the software industry tends to be particularly bad about this pendulum swinging back and forth between ext…
> the pendulum swings too far in one direction, it then swings too far in the other direction
> Hopefully we'll arrive at a reasonable middle ground at some point, and unit tests will be valued (and prioritized) neither too much nor too little.
My coding style has very much gone through a similar evolution. When I first learned about TDD, I went in whole hog - test everything, ui tests, request tests, unit tests - tests for everything.
Then I started to notice that the development costs associated with extreme coverage did not, in fact, pay off.
The number of bugs found was vastly outnumbered by the time wasted dealing with the peculiarities of various ui testing frameworks, let alone the amount of time wasted waiting for those tests to run.
My metric is now that I've written enough tests to feel confident that the code works. It's an extraordinarily qualitative metric, and I cannot find a way to objectively quantify it, yet it very much works for me.
Really, it all comes down to visibility. Is there somewhere that is up to date, that will tell you what the code is supposed to do? Do you have tooling in production that will alert you when it's not doing what it's supposed to do? Do you feel confident that the code works?
Re: Why Most Unit Testing Is Waste [pdf]
#136Earlier quoted context omitted.
Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Were the failing tests due to John/Jane's correctly coded changes, regressions, or bad code changes? The tests provide no meaningful insight into that - it's still ultimately up to the programmer to make that value judgement based on the understanding of what the…
> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". No, the failing tests indicate something changed, and where it changed - what external behavior of a function or class changed. Is the change the right thing, or a bug? Don't know, but it tells you where to look . That's miles better than "I hope this change do…
> where it changed
If we were discussing integration tests, where the interactions between different methods and modules are validated against the input - I'd agree.
But this is about unit tests, in which case the "where" is limited to the method you're modifying, since it's most likely mocked out in other methods and modules to avoid tight coupling.
> includes things as small as the public interface to a class
If we're talking about internal APIs as well, we can't forget that canonical unit testing and TDD frequently requires monkey patching and dependency injection, which can make for some really nasty internal APIs.
> I fail to see how [provably working code is] "the wrong thing".
So, thinking back a bit, I can recall someone showing me TDD, and they gave me the classical example of "how to TDD": Write your first test - that you can call the function. Now test that it returns a number (we were using Python). Now test that it accepts two numbers in. Now test that the output is the sum of the inputs. Congratulations, you're done!
Except you're not, not really. What happens when maxint is one of your parameters? minint? 0? 1? -1? maxuint? A float? A float near to, but not quite 0? infinity? -infinity?
Provably working code is meaningless. Code that can be proven to meet the functionality required of it - that's what you really want. But that's hard to encapsulate in a slogan (like "Red, Green, Refactor"), and harder use as a source of quick dopamine hits.
Re: Why Most Unit Testing Is Waste [pdf]
#137Earlier quoted context omitted.
> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". No, the failing tests indicate something changed, and where it changed - what external behavior of a function or class changed. Is the change the right thing, or a bug? Don't know, but it tells you where to look . That's miles better than "I hope this change do…
> where it changed If we were discussing integration tests, where the interactions between different methods and modules are validated against the input - I'd agree. But this is about unit tests, in which case the "where" is limited to the method you're modifying, since it's most likely mocked out in other methods and modules to avoid tight coupling. > includes things as small as the public interface to a class If we…
> But this is about unit tests, in which case the "where" is limited to the method you're modifying...
Sure, but the consequences may not be. Here's a class where you have two public functions, A and B. Both call an internal function, C. You're trying to change the behavior of A because of a bug, or a requirement change, or whatever. In the process, you have to change C. The unit tests show that B is now broken. Sure, it's the change to C that is at fault, but without the tests, it's easy to not think of checking B. That's the "where" that the tests point you to.
> Provably working code is meaningless. Code that can be proven to meet the functionality required of it - that's what you really want.
Um, yeah, of course that's what you really want. Maybe you should write your tests for that, and not for stuff you don't want? If you don't care about maxint (and you're pretty sure you're never going to), don't write a test for maxint. But it might be worth taking a minute to think about whether you actually do need to handle it (and therefore test for it).
Re: Why Most Unit Testing Is Waste [pdf]
#138Earlier quoted context omitted.
Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Were the failing tests due to John/Jane's correctly coded changes, regressions, or bad code changes? The tests provide no meaningful insight into that - it's still ultimately up to the programmer to make that value judgement based on the understanding of what the…
>> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Even if all you know is "something changed", that's valuable. Pre-existing unit tests can give you confidence that you understand the change you made. You may find an unexpected failure that alerts you to an interaction you didn't consider. Or maybe you'll see…
Agreed, testing is an art that most developers do not have. It must be cultivated and honed. I have been programming for over 35 years and developing tests is painstakingly difficult to cover all your bases.
Re: Why Most Unit Testing Is Waste [pdf]
#139Earlier quoted context omitted.
> ... if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Without getting into the greater unit testing debate... the point would be having a testable code base. Those who test-first guarantee that tests exist and that code can be exercised in a test-harness. Those who do not generally have code that cannot be exe…
Testable code is not a good thing in and of itself. Code should only be testable at its high level interfaces i.e. interfaces that are close to the requirements. So if the code is a library then the high level interface may be the public interface of a class. In that case you may need single class tests. But to do anything useful (in the sense of high level requirements) most code use a combination of various differe…
> Testable code is not a good thing in and of itself. Code should only be testable at its high level interfaces i.e. interfaces that are close to the requirements. So if the code is a library then the high level interface may be the public interface of a class.
Perhaps I misunderstand you, but I have to disagree with this.
Testable code is absolutely a good thing in and of itself.
When something breaks, being able to isolate each piece of the codebase and figure out why it's breaking is essential.
Far too often I've got bug reports from other developers saying "System X is broken, what changed?". When I ask for more detail, a test case proving it's broken, etc - they can't come up with that. Their test case is "our entire service is broken and I think the last thing that happened was related to X". Sometimes it is a problem with System X, but more frequently it's something else in their application.
The smaller you can make your test case, the better, and the easier it is to get a grasp of the acutal problem.
Re: Why Most Unit Testing Is Waste [pdf]
#140Earlier quoted context omitted.
Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Were the failing tests due to John/Jane's correctly coded changes, regressions, or bad code changes? The tests provide no meaningful insight into that - it's still ultimately up to the programmer to make that value judgement based on the understanding of what the…
>> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Even if all you know is "something changed", that's valuable. Pre-existing unit tests can give you confidence that you understand the change you made. You may find an unexpected failure that alerts you to an interaction you didn't consider. Or maybe you'll see…
> If there's 1 thing most proponents and opponents of unit testing can agree on, it's probably that low-value tests are worse than none at all.
Err...just do a quick search in the comments and you will see that this is hardly the consensus.