Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Most Unit Testing Is Waste [pdf]

rbcs-us.com

Post reply on HN
Print view View original

71–80 of 159 posts

Re: Why Most Unit Testing Is Waste [pdf]

#71

Post by mempko »

BlackFly wrote: post #44
lmm wrote: post #28

Earlier quoted context omitted.

Side effects are by definition outside the thing itself, so they're not really amenable to unit testing; you have to use integration tests.

class AccessCountedInteger { private final int value; private int accessCount = 0; AccessCountedInteger(int value) { this.value = value; } public int getValue() { accessCount++; return value; } public int getAccessCount() { return accessCount; } } This unit has side effects on the method getValue() that impact what is returned by getAccessCount(). While it is contrived, if you are making something following the build…

Why in the hell would you write a unit test in your contrived example? I mean look at it!

Re: Why Most Unit Testing Is Waste [pdf]

#72

Post by mannykannot »

ekidd wrote: post #55
mannykannot wrote: post #37

Earlier quoted context omitted.

The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value. In reality, this approach would be useless even for functionally-invisible refactoring or rewriting, precisely because it would be at the unit level. For t…

> The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value. Actually, this is a surprisingly useful strategy! There are two common versions of this idea: 1. Invariant testing with random data. This usually goes by…

These are indeed useful strategies, but they do not invalidate the point I am making here. Fuzz testing is system-level testing, not unit tests, and so is along the lines Coplien is proposing (especially when combined with assertions in the code.) Invariant testing implies an invariant, i.e. an interface-level design contract, while the post I was replying to was predicated on systems where "the code is the spec", and my reply considers only automated unit-test generation that is based only on what the code does, not what it is supposed, in more abstract terms, to do. The "logical conclusion" is based on the possibility of automatically generating unit test cases that function as such, up to any given level of coverage, without adding value.

Re: Why Most Unit Testing Is Waste [pdf]

#73

Post by discreteevent »

bonesss wrote: post #30
mattmanser wrote: post #21

Earlier quoted context omitted.

The article addresses your first point in a big section: 1.4 The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process Your other point about only good programmers not needing unit tests is moot as you haven't followed it through to the conclusion. Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. S…

> ... if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Without getting into the greater unit testing debate... the point would be having a testable code base. Those who test-first guarantee that tests exist and that code can be exercised in a test-harness. Those who do not generally have code that cannot be exe…

Testable code is not a good thing in and of itself. Code should only be testable at its high level interfaces i.e. interfaces that are close to the requirements. So if the code is a library then the high level interface may be the public interface of a class. In that case you may need single class tests. But to do anything useful (in the sense of high level requirements) most code use a combination of various different classes or modules. Write tests at this level because it gives you two advantages:

1) If you decide to change your implementation (refactor low-level code to make it more performant/re-usable/maintainable) then you won't have to change your tests.

2) It will simplify your implementation code.

The second point is as important as the first: In general your code should have the minimum number of abstractions needed to satisfy your requirements (including requirements for re-use/flexibility etc). Testable code often introduces new abstractions or indirections just for the sake of making it possible to plug in a test. One or two of these are OK. But when you add up all the plug points for low level tests in a system it can make the code a lot more complex and indirect than it needs to be. This actually makes it harder to change which was not the idea behind making it testable in the first place.

Gui code may be an exception here. Because gui's are often slow and unstable to test it is worth cleanly separating the gui from the model just to make the system more testable, even if you never intend to use the model with any other kind of ui (the usual motivator for separating model and gui). Any model-view system will help here. If you have a system where the gui is more or less a pure function of the model then you can write all your scenario/stateful tests at the model level and just write a few tests for the ui to verify that it reflects the model accurately

Re: Why Most Unit Testing Is Waste [pdf]

#74

Post by sloppycee »

ekidd wrote: post #54
mattmanser wrote: post #21

Earlier quoted context omitted.

The article addresses your first point in a big section: 1.4 The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process Your other point about only good programmers not needing unit tests is moot as you haven't followed it through to the conclusion. Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. S…

> Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Let's assume you hire good programmers, because otherwise you're doomed. But oftentimes, the "good" programmer and the "bad" programmer are the same person, six months apart: 1. John writes some good code, with good integration tests and good unit tests…

Great points, especially:

> Jane's an awesome developer, but she just got dropped into 20,000 lines of unfamiliar code. The tests will help ensure she doesn't break too much.

In my experience, a well tested codebase lowers the risk and decreases the time to productivity of new hires.

Re: Why Most Unit Testing Is Waste [pdf]

#75

Post by mattmanser »

ekidd wrote: post #54
mattmanser wrote: post #21

Earlier quoted context omitted.

The article addresses your first point in a big section: 1.4 The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process Your other point about only good programmers not needing unit tests is moot as you haven't followed it through to the conclusion. Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. S…

> Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Let's assume you hire good programmers, because otherwise you're doomed. But oftentimes, the "good" programmer and the "bad" programmer are the same person, six months apart: 1. John writes some good code, with good integration tests and good unit tests…

I appreciate your answer, but the destructiveness of the reward loop is directly addressed in the PDF.

I also have no problems designing APIs, experience is what you need and the experience in asking the right questions. No amount of TDD will solve you getting half-way through a design and then finding you needed many-many because you misunderstood requirements.

Other than that API design is (fairly) trivial. I spend a tiny amount of my overall programming time on it. I will map out entire classes and components for large chunks of functionality without filling in any of the code in less than an hour or two and without really thinking about it. Then the hard part of writing the code begins which might take a month or two, and those data structures and API designs will barely change. I see my colleagues doing similar check-ins, un-fleshed components with the broad strokes mapped out.

I'd say it's similar to how we never talk about SQL normal forms any more. 15 years ago the boards would discuss it at length. Today no-one seems to. Why? Because we understand it and all the junior programmers just copy what we do (without realizing it took us a decade to get it right). API design was hard, but today it's not, we know what works and everyone uses it. We've learnt from Java Date and the vagaries in .net 1.0 or PHP's crazy param ordering and all the other difficult API designs from the past and now just copy what everyone else does.

Re: Why Most Unit Testing Is Waste [pdf]

#76

Post by taneq »

mattmanser wrote: post #21
alkonaut wrote: post #10

The normal practice for large scale codebases in complex domains is "the code is the spec". That is, the only specification for how the system should work is how it worked yesterday. In that case, unit tests serve as a great specification. Even tests that just duplicate the business code under test and assert that it's the same (A huge waste in normal cases) is useful. Because a Unit test is much better than a word d…

The article addresses your first point in a big section: 1.4 The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process Your other point about only good programmers not needing unit tests is moot as you haven't followed it through to the conclusion. Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. S…

Tests don't have to be smarter than the code. They just have to be different code. They're screening tests, not diagnostic tests - if the test and the code disagree, you might have a problem. If they don't, then hopefully you don't.

Re: Why Most Unit Testing Is Waste [pdf]

#77

Post by ballenf »

vog wrote: post #34
fermigier wrote: post #24

Earlier quoted context omitted.

Also, there's a followup by Coplien which has not been discussed: https://rbcs-us.com/documents/Segue.pdf So maybe this document should be the focus of the discussion now.

I'd love to submit this document to HN, so it has a chance to get a wider audience than provided by this sub thread. However, the title "Seque" is completely unspecific and hence totally useless. Given that title, I don't see how to submit this to HN in a way that it attracts readers. This is really a pity.

The guidelines on titles allow for writing your own title as long as accurate and more informative than the original title.

The bigger problem, I think, is the velocity of articles appearing and disappearing from the front page makes follow-up articles have an inherently smaller starting audience.

Re: Why Most Unit Testing Is Waste [pdf]

#78

Post by DanBC »

ballenf wrote: post #77
vog wrote: post #34

Earlier quoted context omitted.

I'd love to submit this document to HN, so it has a chance to get a wider audience than provided by this sub thread. However, the title "Seque" is completely unspecific and hence totally useless. Given that title, I don't see how to submit this to HN in a way that it attracts readers. This is really a pity.

The guidelines on titles allow for writing your own title as long as accurate and more informative than the original title. The bigger problem, I think, is the velocity of articles appearing and disappearing from the front page makes follow-up articles have an inherently smaller starting audience.

> The guidelines on titles allow for writing your own title as long as accurate and more informative than the original title.

Mods appear really clear about this. They want you to use the original title unless it's clickbait, or inflammatory, or misleading. Only then can you change it, and they say an informative sentence from the article might do.

This means that articles with terrible, unclear, titles get posted to HN all the time.

See eg this thread: https://news.ycombinator.com/item?id=15540388

(Of course, I'm not a mod, so maybe I'm wrong.)

Re: Why Most Unit Testing Is Waste [pdf]

#79

Post by ekidd »

mannykannot wrote: post #72
ekidd wrote: post #55

Earlier quoted context omitted.

> The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value. Actually, this is a surprisingly useful strategy! There are two common versions of this idea: 1. Invariant testing with random data. This usually goes by…

These are indeed useful strategies, but they do not invalidate the point I am making here. Fuzz testing is system-level testing, not unit tests, and so is along the lines Coplien is proposing (especially when combined with assertions in the code.) Invariant testing implies an invariant, i.e. an interface-level design contract, while the post I was replying to was predicated on systems where "the code is the spec", an…

OK, just to back up a step, here's my best argument in favor of automated testing:

Virtually nobody just makes a change and ships it users without some kind of testing. Maybe you run your program by hand and mess around with the UI. Maybe you call the modified function from a listener with a few arguments. Maybe you have sadistic paid testers who spend 8 weeks hammering on each version before you print DVDs. (And those testers will inevitably write a "test plan" containing things to try with each new release.)

The goal of automated testing is to take all those things you were going to do anyway (calling functions, messing with the UI, following a test plan), and automate them. Different kinds of testing result in different kinds of tests:

1. Manually calling a function can be replaced by unit tests, or even better, doc tests. Everybody loves accurate documentation showing how to use an API!

2. Manually messing around with the UI can be replaced by integration tests.

3. Certain things are much faster to test thoroughly in isolation, so you may mock out parts of your system while testing other parts in great detail, then tie everything together with a few system-wide tests.

People love to invent elaborate taxonomies of testing (unit, request, system, integration, property, etc.). But really, they're all just variations on the same idea: You'd never ship code without testing it somehow, and it's boring and error-prone to do all that testing by hand every time. So automate it!

Re: Why Most Unit Testing Is Waste [pdf]

#80

Post by dkarl »

Don’t underestimate the intelligence of your people, but don’t underestimate the collective stupidity of many people working together in a complex domain.

Sometimes I find that unit testing is an attempt to spend as much time as possible experiencing the first reality (you are a smart, competent programmer) as a relief from the pain of the second reality (you have a gnawing fear that your understanding of the context is fatally flawed.) Can't figure out if the work you're doing is constructive? Anxious about the lack of requirements? Go soothe yourself writing unit tests. Kick back and watch the integration tests run. You're doing your job, and the rest will take care of itself, right?

Post reply on HN