Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Most Unit Testing Is Waste [pdf]

rbcs-us.com

Post reply on HN
Print view View original

51–60 of 159 posts

Re: Why Most Unit Testing Is Waste [pdf]

#51

Post by mannykannot »

alkonaut wrote: post #41
mannykannot wrote: post #37

Earlier quoted context omitted.

The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value. In reality, this approach would be useless even for functionally-invisible refactoring or rewriting, precisely because it would be at the unit level. For t…

> The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value. I don't think that conclusion is any more valid than the proposition that if we just skipped the developer alltogether and generated both the business co…

> I don't think that conclusion is any more valid than the proposition that if we just skipped the developer alltogether and generated both the business code AND the test, we'd have something of value.

That would have some value, as evidenced by all the useful spreadsheets produced by end users, but the the value of the conjunction ("the business code AND the test") lies entirely in the 'business code' part, which is why we do not automatically generate unit tests for spreadsheet macros, or any other code for that matter.

>I think higher level tests are good, but I don't think that makes lower level tests bad.

That's not my point; my point is that unit tests are not automatically useful. Once we accept that, Coplien's argument that there are better ways to spend our limited resources cannot be dismissed (and resources are always limited, because of the combinatorial explosion, which is most evident at the unit level).

Re: Why Most Unit Testing Is Waste [pdf]

#52

Post by lmm »

bonesss wrote: post #42
lmm wrote: post #28

Earlier quoted context omitted.

Side effects are by definition outside the thing itself, so they're not really amenable to unit testing; you have to use integration tests.

Just... small point, but: side effects can be encapsulated, abstracted, and injected to create testable code in "side-effect" heavy situations. Complex running dialog with a hardware serial device, for example, could be rearchitected as a component that has a clear-text dialog with a serial device proxy. This would create functional, side-effect free, testable units of code while also maintaining rich side effects in…

> side effects can be encapsulated, abstracted, and injected to create testable code in "side-effect" heavy situations.

Absolutely, but at that point you're moving the side effect out to the boundary and separating it from the logic. Then you can unit-test the logic away from the side effect (but the test of the side effect itself still has to be an integration test).

Re: Why Most Unit Testing Is Waste [pdf]

#53

Post by lmm »

BlackFly wrote: post #44
lmm wrote: post #28

Earlier quoted context omitted.

Side effects are by definition outside the thing itself, so they're not really amenable to unit testing; you have to use integration tests.

class AccessCountedInteger { private final int value; private int accessCount = 0; AccessCountedInteger(int value) { this.value = value; } public int getValue() { accessCount++; return value; } public int getAccessCount() { return accessCount; } } This unit has side effects on the method getValue() that impact what is returned by getAccessCount(). While it is contrived, if you are making something following the build…

That is not the kind of side effect that Haskellers would acknowledge as necessary (and I'd see it as a bad idea). We can call it a "side effect" but I think there's a qualitative difference between that and something like file I/O or async.

Re: Why Most Unit Testing Is Waste [pdf]

#54

Post by ekidd »

mattmanser wrote: post #21
alkonaut wrote: post #10

The normal practice for large scale codebases in complex domains is "the code is the spec". That is, the only specification for how the system should work is how it worked yesterday. In that case, unit tests serve as a great specification. Even tests that just duplicate the business code under test and assert that it's the same (A huge waste in normal cases) is useful. Because a Unit test is much better than a word d…

The article addresses your first point in a big section: 1.4 The Belief that Tests are Smarter than Code Telegraphs Latent Fear or a Bad Process Your other point about only good programmers not needing unit tests is moot as you haven't followed it through to the conclusion. Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. S…

> Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point?

Let's assume you hire good programmers, because otherwise you're doomed. But oftentimes, the "good" programmer and the "bad" programmer are the same person, six months apart:

1. John writes some good code, with good integration tests and good unit tests. He understands the code base. When he deploys his code, he finds a couple of bugs and adds regression tests.

2. Six months later, John needs to work on his code again to replace a low-level module. He's forgotten a lot of details. He makes some changes, but he's forgotten some corner cases. The tests fail, showing him what needs to be fixed.

3. A year later, John is busy on another project, and Jane needs to take over John's code and make significant changes. Jane's an awesome developer, but she just got dropped into 20,000 lines of unfamiliar code. The tests will help ensure she doesn't break too much.

Also, unit tests (and specifically TDD) can offer two additional advantages:

1. They encourage you to design your APIs before implementing them, making APIs a bit more pleasant and easier to use in isolation.

2. The "red-green-refactor-repeat" loop is almost like the "reward loop" in a video game. By offering small goals and frequent victories, it makes it easier to keep productivity high for hours at a time.

Sometimes you can get away without tests: smaller projects, smaller teams, statically-typed languages, and minimal maintenance can all help. But when things involve multiple good developers working for years, tests can really help.

Re: Why Most Unit Testing Is Waste [pdf]

#55

Post by ekidd »

mannykannot wrote: post #37
alkonaut wrote: post #27

Earlier quoted context omitted.

My point was that even bad unit tests are better than both good/bad word docs or no unit tests. So even bad tests written by bad programmers have a value. They have a large COST too (which is what he's arguing), my argument was merely that the value might not be less than the cost, which is his assertion. > Software engineering research has shown that the most costeffective > places to remove bugs are during the tran…

The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value. In reality, this approach would be useless even for functionally-invisible refactoring or rewriting, precisely because it would be at the unit level. For t…

> The logical conclusion of this line of argument, which is dependent on the proposition that even bad unit tests have value, is that we could just automatically generate copious numbers of unit test cases for a given code base, and we would have something of value.

Actually, this is a surprisingly useful strategy! There are two common versions of this idea:

1. Invariant testing with random data. This usually goes by the name of "QuickCheck", and it allows writing tests that say things like, "If we reverse a random list twice, then it should equal the original list." In practice, this catches tons of corner case bugs.

2. Fuzz testing. In this case, the invariant is, "No matter how broken the input, my program should never crash (or corrupt the heap, or access uninitialized memory, or whatever)." Then you generate a billion random test cases and see what happens. This usually finds tons of bugs, and it's a staple of modern security testing.

So yeah, even random test cases are very valuable. :-)

Re: Why Most Unit Testing Is Waste [pdf]

#56

Post by blub »

alfiedotwtf wrote: post #14
alkonaut wrote: post #10

The normal practice for large scale codebases in complex domains is "the code is the spec". That is, the only specification for how the system should work is how it worked yesterday. In that case, unit tests serve as a great specification. Even tests that just duplicate the business code under test and assert that it's the same (A huge waste in normal cases) is useful. Because a Unit test is much better than a word d…

This. We tend to put a high value on code review, yet some don't value unit testing. Lean in and I'll tell you a secret: ... unit testing is automated, repeatable, code review!

I don't see such a large overlap between code review and unit testing. Here's an example from one of my code reviews, where I discovered two issues:

1) A relaxed requirement was agreed to by the PO, but the remote developer wasn't aware of that and wrote code to fulfil the original complex requirement. The implementation was harder to understand and touched more areas of the code, leading to issue number 2.

2) By analysing the interactions between multiple units, I was able to determine that the code as implemented was in fact reading a setting too early, when it was not available, thereby having no effect at all compared to the already existing code. Ironically, this exact concern had prompted the negotiation with the PO which resulted in the relaxed requirement.

Interestingly the error was not caught by unit tests, because it didn't happen in a single unit. It wasn't caught by integration tests, because that particular online component was mocked and it passed code review by two other developers.

Re: Why Most Unit Testing Is Waste [pdf]

#58

Post by mannykannot »

juandazapata wrote: post #50
wtetzner wrote: post #49

Earlier quoted context omitted.

Refactoring code usually means you have to refractor unit tests as well, since they are so closely tied to the structure of the code. Integration tests, however, should not have to change, since they are typically run against the external interface to the software, which shouldn't change if you're just doing a refactoring.

Funny thing. Unit tests force you to think about good code design. Integration tests don't. If you only have integration tests in your system, you'll most likely end up with a big ball of mud. I've seen this so many times in so many real world projects that it hurts.

> Funny thing. Unit tests force you to think about good code design. Integration tests don't.

They absolutely do, but at higher levels of abstraction, in terms of interfaces, interactions, constraints, obligations and responsibilities. In fact, unit and integration tests make you think about design in essentially the same way.

Re: Why Most Unit Testing Is Waste [pdf]

#59

Post by nradov »

mannykannot wrote: post #40
alfiedotwtf wrote: post #14

Earlier quoted context omitted.

This. We tend to put a high value on code review, yet some don't value unit testing. Lean in and I'll tell you a secret: ... unit testing is automated, repeatable, code review!

How many times do you need to repeat the review of a given unit of code? (actually, that would have more chance of catching additional bugs than would rerunning the same unit tests.) Once you change the code, the existing unit tests are, by definition, invalidated (i.e. there is no value to being able to rerun them), while most higher-level testing will retain their validity. You have actually created an argument for…

No, changing the code doesn't invalidate the unit tests unless the API design contact changes.

Re: Why Most Unit Testing Is Waste [pdf]

#60

Post by pqwEfkvjs »

wtetzner wrote: post #48
pqwEfkvjs wrote: post #38

"Unit tests are pointless" -- this is often said by developers who suffer Dunning-Kruger syndrome. Except the devs who are actually working on something more exotic that cannot be tested well automatically.

You can have automated integration tests.

Yes, but devs who don't write unit tests are probably not going to write integration or acceptance tests either.

Maybe except of a one guy who I talked to a while ago. He does not write unit tests, because static type checking in C++ takes care of everything that unit tests do (according to him), but I actually have seen his code that contained few system tests, so there are exceptions.

Post reply on HN