Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Most Unit Testing Is Waste [pdf]

rbcs-us.com

Post reply on HN
Print view View original

121–130 of 159 posts

Re: Why Most Unit Testing Is Waste [pdf]

#121

Post by njharman »

derefr wrote: post #120
alkonaut wrote: post #10

The normal practice for large scale codebases in complex domains is "the code is the spec". That is, the only specification for how the system should work is how it worked yesterday. In that case, unit tests serve as a great specification. Even tests that just duplicate the business code under test and assert that it's the same (A huge waste in normal cases) is useful. Because a Unit test is much better than a word d…

> That is, the only specification for how the system should work is how it worked yesterday. You'd think, if this was the desired model, you could partially automate the "writing unit tests" part of this process. (Integration tests no, but unit tests yes.) The "spec" for the unit tests is already there, in the form of the worktree of the previous, known-good commit. That means that, in a dynamic language, you'd just…

> The "spec" for the unit tests is already there

Ah, but the spec is often really poorly defined. That is all sorts of edge cases and uncommon code paths do not work as imagined. And, of course, it is full of bugs.

I have found countless "bugs" and issues when writing tests for existing code. I also often have to refactor the code to make it clearer what it's doing or to make it testable.

Unit tests are a development tool more than they are a testing tool. They are a means to produce correct, well documented, well speced, cleanly separated code.

Re: Why Most Unit Testing Is Waste [pdf]

#123

Post by lewisl9029 »

h0l0cube wrote: post #101

People seem to have not read the article properly and are conflating unit testing, with all testing. The article doesn't say to avoid all unit testing, but to restrict the test suite to that which can be validated by business logic or some formalized oracle - this would imply unit testing for critical systems like hardware drivers, banking/avionics, crypto, etc, where there are known and eternally consistent results…

Agreed on all points.

These days, a large majority of the frontend tests in my codebase aren't tests that I've written myself. They're automatically generated Jest snapshot tests that capture the virtual DOM output of entire components in different states as dictated by product requirements, using the StoryShots plugin for React Storybook.

These tests prove to be extremely effective in catching bugs, because they directly reflect product requirements, and because they end up exercising a large majority of the codebase for very little cost compared to covering the same amount of code with individual unit tests.

The latter point is an important consideration too, because a lot of the code these integration-level tests exercise are code that I'd have considered too trivial to have been worth the ongoing maintenance costs in building unit tests for, when considered in isolation.

This means it'd have been easy for me to be satisfied with selectively and arbitrarily deciding which pieces of code should qualify for test coverage, and leave test coverage at some arbitrary number, as opposed to what I do now, which is to always strictly require 100% code coverage, but explicitly and deliberately exclude code that I have good reason to not test, which feels like a much more solid framework for ensuring code quality.

I still write unit tests for functionality that component snapshot tests can't reasonably cover, but usually only at a level of granularity where the tests themselves can map cleanly to product requirements as well, instead of painstakingly testing every little function in perfect isolation.

Re: Why Most Unit Testing Is Waste [pdf]

#124

Post by groby_b »

> 1.4 The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process

That's not why you write unit tests. They are, in a roundabout way, programming's way of double-entry bookkeeping. Tests and code both are there to double check the other one. Neither one is "smarter" than the other.

Re: Why Most Unit Testing Is Waste [pdf]

#125

Post by lolsal »

fermigier wrote: post #23

This has already been discussed to death, on 3 occasions: https://news.ycombinator.com/item?id=7353767 (268 comments) https://news.ycombinator.com/item?id=11799272 (280 comments) https://news.ycombinator.com/item?id=13815779 ("only" 24 comments)

I had not seen this before. I appreciate it being posted.

Re: Why Most Unit Testing Is Waste [pdf]

#126

Post by jblow »

Const-me wrote: post #61
juandazapata wrote: post #20

It'll be fun to refactor a codebase without unit tests

Did that many times without problems. But, I mostly code C++ and C#, both are strongly typed, so the compiler helps a lot with refactors.

Exactly.

Most proponents of unit tests use horrible programming languages; they are afraid to change code because anything could break at any time. Stop using those languages and most of the problems 'fixed' by unit testing just disappear.

Re: Why Most Unit Testing Is Waste [pdf]

#127

Post by jrumbut »

ekidd wrote: post #88
mattmanser wrote: post #75

Earlier quoted context omitted.

I appreciate your answer, but the destructiveness of the reward loop is directly addressed in the PDF. I also have no problems designing APIs, experience is what you need and the experience in asking the right questions. No amount of TDD will solve you getting half-way through a design and then finding you needed many-many because you misunderstood requirements. Other than that API design is (fairly) trivial. I spend…

> I appreciate your answer, but the destructiveness of the reward loop is directly addressed in the PDF. I've gone back through most of the PDF, and I can't figure out what section you're referring to. There's a bunch of discussion of perverse incentives (mostly involved incompetent managers or painfully sloppy developers, who will fail with any methodology), but I don't see where the author addresses what I'm talkin…

Your comments on this thread have really helped clarify my thinking on this important issue.

Re: Why Most Unit Testing Is Waste [pdf]

#128

Post by AnimalMuppet »

falcolas wrote: post #109
ekidd wrote: post #54

Earlier quoted context omitted.

> Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Let's assume you hire good programmers, because otherwise you're doomed. But oftentimes, the "good" programmer and the "bad" programmer are the same person, six months apart: 1. John writes some good code, with good integration tests and good unit tests…

Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Were the failing tests due to John/Jane's correctly coded changes, regressions, or bad code changes? The tests provide no meaningful insight into that - it's still ultimately up to the programmer to make that value judgement based on the understanding of what the…

> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed".

No, the failing tests indicate something changed, and where it changed - what external behavior of a function or class changed. Is the change the right thing, or a bug? Don't know, but it tells you where to look. That's miles better than "I hope this change doesn't break anything".

> "design your APIs before implementing them" - this only works out if we know our requirements ahead of time.

No. "API" here includes things as small as the public interface to a class, even if the class is never used by anything other than other classes in the module. You have some idea of the requirements for the class at the time when you're writing the class; otherwise, you have no idea what to write! But writing the test makes you think like a user of that class, not like the author of that class. That gives you a chance to see places where the public interface is awkward - places that you wouldn't see as the class author.

> This tells me that we're rewarding the wrong thing: the creation of passing tests, not the creation of correct code.

We're rewarding the creation of provably working code. I fail to see how that's "the wrong thing".

Re: Why Most Unit Testing Is Waste [pdf]

#129

Post by hinkley »

discreteevent wrote: post #73
bonesss wrote: post #30

Earlier quoted context omitted.

> ... if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Without getting into the greater unit testing debate... the point would be having a testable code base. Those who test-first guarantee that tests exist and that code can be exercised in a test-harness. Those who do not generally have code that cannot be exe…

Testable code is not a good thing in and of itself. Code should only be testable at its high level interfaces i.e. interfaces that are close to the requirements. So if the code is a library then the high level interface may be the public interface of a class. In that case you may need single class tests. But to do anything useful (in the sense of high level requirements) most code use a combination of various differe…

The cost of modifying simple tests is negligible. Getting your tests to that point is a big shift in how people interact with them.

Modifying complicated tests can be unbounded, and I think a lot of the pushback comes from these situation.

It is very possible to create tests that are worse than doing nothing. It can be a roadmap for learned helplessness, too.

Re: Why Most Unit Testing Is Waste [pdf]

#130

Post by luord »

> In most businesses, the only tests that have business value are those that are derived from business requirements.

This is, imo, the most important takeaway and it is, or should be, obvious, but it often isn't (maybe because "common sense is the least common of senses" or something like that). It's also what I strive for, with one caveat[1].

> One is to use it as a learning tool: to learn more about the program and how it works.

Another great takeaway. Good tests should work as documentation, imo. I consider this another test quality metric, even: If looking at the tests only confuses me further, those tests need to be changed (and the code they're testing too, most likely).

[1]: Striving for this can turn code coverage into an useful metric. Not of correctness, of course, but on how much code we're writing that doesn't solve a business requirement. That code should be refactored away to separate libraries or replaced with third party libraries that already do that. I'm firmly in the "avoid NIH" camp.

Post reply on HN