Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Most Unit Testing Is Waste [pdf]

rbcs-us.com

Post reply on HN
Print view View original

141–150 of 159 posts

Re: Why Most Unit Testing Is Waste [pdf]

#141

Post by fpig »

josteink wrote: post #85
taneq wrote: post #76

Earlier quoted context omitted.

Tests don't have to be smarter than the code. They just have to be different code. They're screening tests, not diagnostic tests - if the test and the code disagree, you might have a problem. If they don't, then hopefully you don't.

Actually I’d argue tests should never be smart code. It should be short, plain, simple and make it obvious what is being tested, how and why. Because when that test fails you want to quickly understand what the problem is. “Smart” (or just too big) tests all detract from this.

But exactly these tests tend to be the useless ones that just pass and don't find bugs.

On 2 separate occasions I had an app with extensive unit tests that seemed to work fine but also seemed to have strange rare bugs.

Both times I wrote a single additional "unit" test that fired up the environment (with mocks, same as the other unit tests) but then acted like a consumer of the API and spammed the environment with random (but not nonsense) calls for several minutes. These tests were quite complex so basically the exact opposite of what you're suggesting.

Not only did I immediately find the bug, but in both cases I found like 10 bugs before I got the test to even pass for the first time.

At the same time all the small unit tests were happily passing. Because they didn't hit edge cases (both in data and in timing) that nobody had thought of.

Re: Why Most Unit Testing Is Waste [pdf]

#142

Post by quickthrower2 »

deliveryninja wrote: post #83

The host of this pdf is a testing consultancy which seems to have posted the same article about 10x already to hacker news. It's just spam

I posted, I have no connection to the consultancy. I thought there were some good arguments in the PDF which I agree with after > 10 years dev experience.

Re: Why Most Unit Testing Is Waste [pdf]

#143

Post by valuearb »

Management keeps asking when we will write unit tests, and I tell them I’m too busy fixing bugs, because each time I fix a bug I

1) examine the entire code base to find any similar problems 2) implement full parameter validation in that code to try to ensure it can’t happen again and 3) add exception logging code so that we will get an immediate error reported directly on the line it occurred if it does.

If I have any extra time left over, I refactor to eliminate code duplication, update/verify/add more parameter and input data validation, and increase code encapsulation.

Then I point out our crash rate has dropped by 70% in the 6 months I’ve been working on the project.

Re: Why Most Unit Testing Is Waste [pdf]

#144

Post by MaulingMonkey »

falcolas wrote: post #136
AnimalMuppet wrote: post #128

Earlier quoted context omitted.

> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". No, the failing tests indicate something changed, and where it changed - what external behavior of a function or class changed. Is the change the right thing, or a bug? Don't know, but it tells you where to look . That's miles better than "I hope this change do…

> where it changed If we were discussing integration tests, where the interactions between different methods and modules are validated against the input - I'd agree. But this is about unit tests, in which case the "where" is limited to the method you're modifying, since it's most likely mocked out in other methods and modules to avoid tight coupling. > includes things as small as the public interface to a class If we…

> If we were discussing integration tests

I feel like you're perhaps defining unit tests to exclude "useful unit tests" by relabeling them. Yes, if you exclude all the useful tests from unit testing, unit testing is useless. Does a unit test suddenly become a regression test if the method it tests contains, say, an unmocked sprintf - which has subtle variations in behavior depending on which standard library you link against? No true ~~scotsman~~ unit test would have conditions that would only be likely to fail in the case of an actual bug?

> But this is about unit tests, in which case the "where" is limited to the method you're modifying

That's still useful. C++ build cycles mean I could have easily touched 20 methods before I have an executable that lets me run my unit tests, telling me which 3 of the 20 I fucked up in is useful. Renaming a single member variable could easily affect that many methods, and refactoring tools are often not perfectly reliable.

Speaking of refactoring, I'm doing pure refactoring decently often - I might try to simplify a method for readability before I make behavior changes to it. Any changes to behavior in this context are unintentional - and if they occur, 9 times out of 10 discover I have a legitimate bug. Even pretty terrible unit tests, written by a monkey just looking to increase code coverage and score that dopamine hit can help here - to say nothing of unit tests that rise to the level of being mediocre.

Further, "where" is not limited to "the method you're modifying". I'm often doing multiplatform work - understanding that "where" is in method M... on platform X in build configuration Y is extremely useful. Even garbage unit tests with no sense of "correctness" to them are again useful here - they least tell me I've got an (almost certainly undesirable) inconsistency in my platform abstractions, likely to lead to platform specific bugs (because reliant code is often initially tested on only one of those platforms for expediency). This lets me eliminate those inconsistencies at my earliest convenience. When writing the code to abstract away platform specific details, these inconsistencies are quite common.

Re: Why Most Unit Testing Is Waste [pdf]

#145

Post by taneq »

fpig wrote: post #141
josteink wrote: post #85

Earlier quoted context omitted.

Actually I’d argue tests should never be smart code. It should be short, plain, simple and make it obvious what is being tested, how and why. Because when that test fails you want to quickly understand what the problem is. “Smart” (or just too big) tests all detract from this.

But exactly these tests tend to be the useless ones that just pass and don't find bugs. On 2 separate occasions I had an app with extensive unit tests that seemed to work fine but also seemed to have strange rare bugs. Both times I wrote a single additional "unit" test that fired up the environment (with mocks, same as the other unit tests) but then acted like a consumer of the API and spammed the environment with ra…

Your API-spammer interface reminds me of the Macintosh "Monkey": https://www.folklore.org/StoryView.py?project=Macintosh&stor...

Re: Why Most Unit Testing Is Waste [pdf]

#146

Post by josteink »

fpig wrote: post #141
josteink wrote: post #85

Earlier quoted context omitted.

Actually I’d argue tests should never be smart code. It should be short, plain, simple and make it obvious what is being tested, how and why. Because when that test fails you want to quickly understand what the problem is. “Smart” (or just too big) tests all detract from this.

But exactly these tests tend to be the useless ones that just pass and don't find bugs. On 2 separate occasions I had an app with extensive unit tests that seemed to work fine but also seemed to have strange rare bugs. Both times I wrote a single additional "unit" test that fired up the environment (with mocks, same as the other unit tests) but then acted like a consumer of the API and spammed the environment with ra…

A test can still be simple while providing garbage edge-case data.

In fact that’s usually one of my default tests to ensure that I have a well-defined behavior even in these cases.

Re: Why Most Unit Testing Is Waste [pdf]

#147

Post by krfsm »

Floegipoky wrote: post #133
falcolas wrote: post #109

Earlier quoted context omitted.

Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Were the failing tests due to John/Jane's correctly coded changes, regressions, or bad code changes? The tests provide no meaningful insight into that - it's still ultimately up to the programmer to make that value judgement based on the understanding of what the…

>> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Even if all you know is "something changed", that's valuable. Pre-existing unit tests can give you confidence that you understand the change you made. You may find an unexpected failure that alerts you to an interaction you didn't consider. Or maybe you'll see…

Somebody could do the TDD movement a great big favour and write a TDD instruction for people who actually already know how to write tests.

There probably are good resources about this somewhere, but compared to the impression of "TDD promotes oceans of tiny, low-value tests" they lack visibility.

Re: Why Most Unit Testing Is Waste [pdf]

#148

Post by ekidd »

falcolas wrote: post #136
AnimalMuppet wrote: post #128

Earlier quoted context omitted.

> Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". No, the failing tests indicate something changed, and where it changed - what external behavior of a function or class changed. Is the change the right thing, or a bug? Don't know, but it tells you where to look . That's miles better than "I hope this change do…

> where it changed If we were discussing integration tests, where the interactions between different methods and modules are validated against the input - I'd agree. But this is about unit tests, in which case the "where" is limited to the method you're modifying, since it's most likely mocked out in other methods and modules to avoid tight coupling. > includes things as small as the public interface to a class If we…

> canonical unit testing and TDD frequently requires monkey patching and dependency injection

This is probably the root of most of our disagreement. I belong to the school of thought that says (in most cases) "Mocking is a Code Smell": https://medium.com/javascript-scene/mocking-is-a-code-smell-... And dependency injection (especially at the unit test level) is giant warning sign that you need to reconsider your entire architecture.

Nearly all unit tests should look like one of:

- "I call this function with these arguments, and I get this result."

- "I construct a nice a little object in isolation, mess with it briefly, and here's what I expect to happen."

At least 50% of the junior developers I've mentored can learn to do this tastefully and productively.

But if you need to install 15 monkey patches and fire up a monster dependency injection framework, something has gone very wrong.

But this school of thought also implies that most unit tests have something in common with "integration" tests—they test a function or a class from the "outside," but that function or class may (as an implementation detail) call other functions or classes. As long as it's not part of the public API, it doesn't need to be mocked. And anything which does need to be mocked should be kept away from the core code, which should be relatively "pure" in a functional sense.

This is more of an old-school approach. I learned my TDD back in the days of Kent Beck and "eXtreme Programming Explained", not from some agile consultant.

Re: Why Most Unit Testing Is Waste [pdf]

#149

Post by Const-me »

pqwEfkvjs wrote: post #60
wtetzner wrote: post #48

Earlier quoted context omitted.

You can have automated integration tests.

Yes, but devs who don't write unit tests are probably not going to write integration or acceptance tests either. Maybe except of a one guy who I talked to a while ago. He does not write unit tests, because static type checking in C++ takes care of everything that unit tests do (according to him), but I actually have seen his code that contained few system tests, so there are exceptions.

> devs who don't write unit tests are probably not going to write integration or acceptance tests either

Why do you think they aren’t? I don’t write unit tests except when I’m specifically paid extra for that (yes, coding strongly-typed languages too), but I do write other kinds of tests as I see fit.

Re: Why Most Unit Testing Is Waste [pdf]

#150

Post by halfnibble »

falcolas wrote: post #109
ekidd wrote: post #54

Earlier quoted context omitted.

> Namely, if bad programmers write bad code that needs unit tests, they're also going to write bad unit tests that don't test the code correctly. So what's the point? Let's assume you hire good programmers, because otherwise you're doomed. But oftentimes, the "good" programmer and the "bad" programmer are the same person, six months apart: 1. John writes some good code, with good integration tests and good unit tests…

Here's the crux of your argument - and I honestly think it's a flawed premise: The failure of unit tests indicates something other than "Something Changed". Were the failing tests due to John/Jane's correctly coded changes, regressions, or bad code changes? The tests provide no meaningful insight into that - it's still ultimately up to the programmer to make that value judgement based on the understanding of what the…

I recently started maintaining a decades old PHP application. I have been selectively placing in unit and functional tests as I go through the massive codebase. Whenever I refactor something I've touched before, I tend to get failing tests. If nothing else, this tells me that some other component I worked on previously is using this module and I should look into how these changes affect it.

Unfortunately, the existing architecture doesn't make dependencies obvious. So simply knowing that "something has changed" is very, very helpful.

Post reply on HN