Why the concern that the volume of tests is greater than the volume of code? Why should that be a constraint, other than some vague sense of wasted time spent typing them out?
Why Most Unit Testing Is Waste [pdf]
111–120 of 159 posts
Re: Why Most Unit Testing Is Waste [pdf]
#112This is truly the dumbest thing I've read all year about software development. It isn't just dumb though, it's harmful. Now moron developers everywhere will hold it up as an excuse why their code is so good it doesn't need testing. Now, especially after Equifax, is a bad time to have this attitude. There are already journalists calling for developer licensure. https://www.nytimes.com/2017/09/11/opinion/equifax-accoun…
This is a critique of popular testing practices by an experienced engineer who is trying to improve the practice of testing and increase software quality. This is a big part of how engineering practices improve. A culture where we call something "dumb" without reading it because we only care what "moron developers" will make of it (presumably also without reading it) isn't going to increase software quality or protect against the next Equifax. It's easy to have a knee-jerk reaction to a title and rationalize the desire to respond on that basis. Surely nobody else will read the article, and therefore I have already determined its true significance by having a knee-jerk response to the title or a few haphazardly skimmed sections. If that reasoning prevailed, though, people should just write headlines and never anything more thoughtful, and we should be stuck forever with "UNIT TESTING GOOD" and "UNIT TESTING BAD" as the two competing pinnacles of software testing wisdom.
In case anyone was fooled by this comment into thinking the article takes a lazy approach to testing, it argues for keeping a certain class of unit tests; for another class, preferring system tests instead; and for third class, turning them into assertions that ship with production code where possible. That's not lazy. Depending on how you currently test your code, it may actually be a harder standard to meet. It's possible you personally have nothing to learn from it, but I would wager there's something in it that will make you see your tests in a different way.
Re: Why Most Unit Testing Is Waste [pdf]
#113I strongly agree, and disagree with many parts of this.
One part I strongly disagree with, is this passage:
> Most programmers want to "hear" the "information" that their program component works. So when they wrote their first function for this project three years ago they wrote a unit test for it. The test has never failed. The question is: How much information is in that test? That is, if "1" is the passing of a test and "0" is the failing of a test, how much information is in this string of test results:
> 11111111111111111111111111111111
> There are several possible answers depending on which formalism you apply, but most of the answers are wrong. The naive answer is 32, but that is the bits of data, not of information.
Just because a test has passed in your Continuous Integration environment 100% of the time, doesn't mean that test is worthless. I have checked in many tests that have never failed in CI - but have failed when I was working on my code. However, since there's no shiny team-visible metrics with bar charts about how often a test failed in a local workspace, people can wrongly assume that a unit test is worthless.
> Now, how many bits of information in this string of test runs?
> 1011011000110101101000110101101
> The answer is... a lot more. Probably 32.
If I see that in our CI environment, the answer is 'this test is nearly-worthless.' The long answer is 'If it's a unit test, there's a race condition, if it's an integration test, there's a race condition, or it's flaking for reasons outside our control.'
> Another client of mine also had too many unit tests. I pointed out to them that this would decrease their velocity, because every change to a function should require a coordinated change to the test. They informed me that they had written their tests in such a way that they didn't have to change the tests when the functionality changed. That of course means that the tests weren't testing the functionality, so whatever they were testing was of little value.
... That's the whole point of blackbox testing. If observed behaviour is not expected to change, neither should the test. If a refactoring forces you to update the test, then, yes, you are testing at the wrong level of abstraction.
This piece could have saved a dozen pages if it just told us to stop testing private methods, and write more integration tests.
Re: Why Most Unit Testing Is Waste [pdf]
#114I'm afraid that the TDD movement went to far. The proponents became so convinced that they were right that they started comparing themselves to people who argued that washing hands was important in early surgery, and that those who questioned it would soon be unemployable in the field. Not those who didn't practice it, those who questioned it. I dealt with some of this, including a rather bullying type who tried to b…
As in many areas of thought, when the pendulum swings too far in one direction, it then swings too far in the other direction. Hopefully we'll arrive at a reasonable middle ground at some point, and unit tests will be valued (and prioritized) neither too much nor too little.
On a side note, unfortunately I think the software industry tends to be particularly bad about this pendulum swinging back and forth between extremes thing. There is so much emphasis on innovation that people are biased toward making radical changes and believing they are going to revolutionize everything. And the culture is such that the more you go out on a limb and push something radical, the more you are respected, because we often value guts more than good judgement.
Re: Why Most Unit Testing Is Waste [pdf]
#115The best workflow I've found is to define an empty function, add a breakpoint, and code the function in a repl. At this point I'm pretty confident the code works for at least a 'happy path' input so I copy the code to the editor and add tests to call it with a variety of other inputs.
Being able to inspect the state of the program in real time is invaluable and gives me a lot of confidence that I understand how the code works. For some reason most programmers I see don't even run their code locally and just use log statements to guess at state when it invariably doesn't work as expected.
Another big problem is test data. I see way too much naive mocking. You really need to exercise your code with data that is as close to real-life input as possible, ideally it's a sanitized version of production data. Other tests are great too (i.e large lists of 'naughty strings') but if you're manually specifying your test data you are a) spending a lot of time doing something that should be automated and b) are only exercising your code with what you think it might see which is usually not good enough.
Re: Why Most Unit Testing Is Waste [pdf]
#116The best workflow I've found is to define an empty function, add a breakpoint, and code the function in a repl. At this point I'm pretty confident the code works for at least a 'happy path' input so I copy the code to the editor and add tests to call it with a variety of other inputs. Being able to inspect the state of the program in real time is invaluable and gives me a lot of confidence that I understand how the c…
Log data is how unit testing started I think, people would log all the output and compare the sheet of actual log output to the sheet of expected log output.
Working in an GUI debugger / REPL is great for visualizing code I often code that way myself, but let's not fool ourselves, it still requires manually setting breakpoints, and pressing keys to step through code. You can't do this for every method after every change, whereas unit testing has that advantage. I do agree with most things mentioned in the article though. A lot people end up writing tests that hit databases and are too slow or fail randomly due to chained state, or are over specified & end up just getting in the way. What it comes down to is you can have good tests or bad tests, and its still entirely subjective just like whether the code itself is good or bad. I recommend the book xUnit test patterns, its basically a bunch of "rules of thumb"
Re: Why Most Unit Testing Is Waste [pdf]
#117The SQLite project seems to have an alternative view: https://www.sqlite.org/testing.html A lot of us would consider SQLite to be high quality and relatively bug-free. The extensive test suite they've built up that exercises each release is a huge reason for it. Of test code quantity , Coplien writes: >If your coders have more lines of unit tests than of code, it probably means one of several things. They may be para…
Is there a reason why someone would throw away unit tests? I've never understood this. Time was taken to write it, test it, upheave the bug, and now we want to remove the safeguards that we spent time/money on? Leaving the possibility for the bug to infest again? Scrapping unit tests is scrapping time.
If you are testing at the wrong layer of abstraction (Your test is tightly coupled to the implementation of your class, as opposed to its observable behavior), then refactoring your code will require refactoring the test.
The correct solution to this is to not throw passing tests out, but to stop doing white-box testing.
(Not to mention that a test that may pass in your CI environment may fail - frequently - in your local workspace.)
Re: Why Most Unit Testing Is Waste [pdf]
#118Somebody should tell them about https://en.wikipedia.org/wiki/Code_coverage#Other_coverage_c... and mutation testing.
Re: Why Most Unit Testing Is Waste [pdf]
#119The normal practice for large scale codebases in complex domains is "the code is the spec". That is, the only specification for how the system should work is how it worked yesterday. In that case, unit tests serve as a great specification. Even tests that just duplicate the business code under test and assert that it's the same (A huge waste in normal cases) is useful. Because a Unit test is much better than a word d…
The article addresses your first point in a big section: 1.4 The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process Your other point about only good programmers not needing unit tests is moot as you haven't followed it through to the conclusion. Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. S…
In my experience, it would be pretty hard to consistently write unit tests that pass and at the same time don't test the code. They might not test the code complete enough, but they will test the code. That's still better than not testing at all, and can always be fixed by adding tests later.
Re: Why Most Unit Testing Is Waste [pdf]
#120The normal practice for large scale codebases in complex domains is "the code is the spec". That is, the only specification for how the system should work is how it worked yesterday. In that case, unit tests serve as a great specification. Even tests that just duplicate the business code under test and assert that it's the same (A huge waste in normal cases) is useful. Because a Unit test is much better than a word d…
> That is, the only specification for how the system should work is how it worked yesterday.
You'd think, if this was the desired model, you could partially automate the "writing unit tests" part of this process. (Integration tests no, but unit tests yes.) The "spec" for the unit tests is already there, in the form of the worktree of the previous, known-good commit.
That means that, in a dynamic language, you'd just need a "test suite" consisting of a series of example calls to functions. No outputs specified, no assertions—just some valid input parameters to let the test-harness call the functions. (In a static language, you wouldn't even need that; the test-harness could act like a fuzzer, generating inputs from each function's domain automatically.)
The tooling would then just compare the outputs of the functions in the known-good-build, to the outputs of the same functions from your worktree. Anywhere they differ is an "assertion failure." You'd have to either fix the code, or add a pragma above the function to specify that the API has changed. (Though, hopefully, such pragmas would be onerous enough to get people to mostly add new API surface for altered functionality, rather than in-place modifying existing guarantees.) A pre-commit hook would then strip the pragmas from the finalized commit. (They would be invalid as of the next commit, after all.)
Interestingly, given such pragmas, the pre-commit hook could also automatically derive a semver tag for the new commit. No pragmas? Patch. Pragmas on functions? Minor version. Pragmas on entire modules? Major version.