Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Where unit testing fails

hmemcpy.com

Post reply on HN
Print view View original

41–50 of 52 posts

Re: Where unit testing fails

#41

Post by bruceboughton »

zumda wrote: post #28
smackay wrote: post #18

Earlier quoted context omitted.

The issue/argument here is not that writing tests is worthless but that in many cases the approach of "if we write enough tests then the code that solves the problem will eventually emerge" is doomed to failure or at least results in an extremely verbose approach to development.

But then I think the approach is the wrong one. In TDD you are not required to write a lot of test and hope that they magically work in the future. Instead a small test to do one thing should be written, and then write enough code to make it pass. For example lets take a simple newsletter signup form with only a field to enter an email address. Now you shouldn't write a test to see what happens when you enter an emai…

Sure, if you need training wheels on your bike...

Re: Where unit testing fails

#42

Post by prodigal_erik »

bruceboughton wrote: post #41
zumda wrote: post #28

Earlier quoted context omitted.

But then I think the approach is the wrong one. In TDD you are not required to write a lot of test and hope that they magically work in the future. Instead a small test to do one thing should be written, and then write enough code to make it pass. For example lets take a simple newsletter signup form with only a field to enter an email address. Now you shouldn't write a test to see what happens when you enter an emai…

Sure, if you need training wheels on your bike...

Which we demonstrably do as an industry. When neither NASA nor Knuth can write 100% correct code, the rest of us can't reasonably expect to pull it off. Human beings doing meticulous work make lots and lots of mistakes, and I welcome any feasible tactic that stops mine from being released. Sure, automating "I got a confirmation after pushing the button" seems hokey on day one (it's not as if you'd otherwise commit and go home before having seen it work) but somebody is eventually going to make a change that unexpectedly breaks it, and I want to know about it.

Re: Where unit testing fails

#43

Post by zts »

berntb wrote: post #39
tychof wrote: post #24

Earlier quoted context omitted.

One benefit to TDD is that you become a user for your API much more quickly, and so your API doesn't need to change as much. TDD suggests 3 steps: 1. Write one test 2. Write the minimal code to make it pass 3. Refactor the code you've got. Try not to think of tests as immutable. You can throw away tests when they don't provide value. Finally, make sure you're using an IDE with refactoring support. That helps immensel…

OK, thanks to you and zumda. Interesting TDD argument. I'll buy a book or something and try again when my present hobby project is done and I have time for experimenting. (IDE refactoring needs a stricter language than I prefer. Also, without an IDE and using my old eyes I have 2 A4 of code + 1 A4 of bash on a 24" monitor...)

If you're looking for book suggestions, I'd recommend "Growing Object-Oriented Software, Guided By Tests" by Steve Freeman and Nat Pryce. It's pretty pragmatic, and the treatment of the subject is pretty thought-provoking (at least, I thought so when I read it).

The examples are all Java, but the accompanying website (http://www.growing-object-oriented-software.com/) has links to reimplementations in several other languages.

Re: Where unit testing fails

#44

Post by jsdalton »

latch wrote: post #29
jsdalton wrote: post #19

Earlier quoted context omitted.

Can you expand a bit on the distinction you're making between mocks and stubs, and why you believe stubs are both underused and useful? Perhaps I'm doing it wrong, but I've found in my experience that bugs I miss in testing increase proportionally with my use of mocks or stubs. Invariably, when I'm forced to stub something and replicate its behavior I miss something subtle that comes back to bite me.

I think there's 2 problem. First, people rely too heavily on mocks/stubs/fakes/(whatever you want to call them). This has gotten better in the past 4 or so years (in my mind, largely because of Rails and it being "acceptable" to hit a DB in a test - though it might actually predate Rails). I think this problem is pretty straightforward to understand (again, especially when you look at the Rails way to test a DB inter…

Thanks, it makes perfect sense and is pretty much why I virtually never use mocks. In fact, my personal philosophy is at odds with most advice I have seen around unit testing, in that I try to avoid the use of even stubs wherever possible. In other words, I try to make my unit tests as much like integration tests as possible. Obvious exceptions are things like third-party APIs, which must be stubbed, but everything else I try to get close to production code.

Re: Where unit testing fails

#45

Post by astral303 »

latch wrote: post #29
jsdalton wrote: post #19

Earlier quoted context omitted.

Can you expand a bit on the distinction you're making between mocks and stubs, and why you believe stubs are both underused and useful? Perhaps I'm doing it wrong, but I've found in my experience that bugs I miss in testing increase proportionally with my use of mocks or stubs. Invariably, when I'm forced to stub something and replicate its behavior I miss something subtle that comes back to bite me.

I think there's 2 problem. First, people rely too heavily on mocks/stubs/fakes/(whatever you want to call them). This has gotten better in the past 4 or so years (in my mind, largely because of Rails and it being "acceptable" to hit a DB in a test - though it might actually predate Rails). I think this problem is pretty straightforward to understand (again, especially when you look at the Rails way to test a DB inter…

I've had similar experience with strict mocks. However, mocks don't need to be strict. In Java, for example, the current mocking tool of choice is Mockito, which returns "nice" mocks that return nulls, empty collections and such.

I can't seem to find a blog post discussing the merits, but IME strict mocking leads to a lot of tests breaking just because you added one more piece of functionality to a method. Now, this causes you to edit all sorts of existing tests to cause them to ignore a call to collaborator. While you can extract common expectations and such to eliminate duplication, you still end up with many tests failing due to one small change in behavior.

Normally your tests tend to assert one or two behaviors per test. With strict mocking, your tests become awkwardly cluttered by setting up expectations/interactions for things that are unrelated to what they are asserting.

With nice mocking, it's much easier to get away with only the expectations/interactions that are directly relevant to what a single test is asserting. This also means that you get more fine-grained failures, which speeds up diagnostics.

Re: Where unit testing fails

#46

Post by jdlshore »

peteretep wrote: post #4

Whut? Listen kids, testing is a tool for HELPING THE DEVELOPER, not for using to engage in a "more pious than thou" dick-swinging my Cucumber is bigger than yours idiocy. Testing is about giving YOU THE DEVELOPER useful and quick feedback about if you're on the right path, and if you've broken something, and for warning people who come after you if they've broken something. It's not an arcane methodology that somehow…

I'm seeing a lot of misconceptions about test-driven development in this thread. The biggest one is that test-driven development involves writing a bunch of tests first. Instead, TDD involves interweaving tests and production code in a very tight cycle. You write a few lines of test code, then a few lines of production code, then you review and improve your code, then write a few lines of test code, and so forth. Each step only takes a minute or so.

This is useful because you're constantly making and checking hypotheses: "This code should make a test fail in X way... yup. Okay, now this code should make the test pass. Yup. Now this change shouldn't change behavior, and the tests should still pass. Oops! Undo. Oh, that method has to be called in the constructor. All right, this change should work. Good."

(If you're interested, my book goes into more detail, and it's online: http://jamesshore.com/Agile-Book/test_driven_development.htm... . I also have a extensive screencast of real-world TDD here: http://jamesshore.com/Blog/Lets-Play/ )

Despite the extremists, TDD really is a useful tool--and that's all it is. Sure, TDD has been turned into yet another vehicle for programmers to engage in religious wars, but you can't really blame TDD for that, any more than you can blame Richard Stallman for emacs vs. vi wars.

Re: Where unit testing fails

#47

Post by Volpe »

peteretep wrote: post #36
Volpe wrote: post #14

Earlier quoted context omitted.

> Writing tests before code works pretty well in some situations. Test Driven Development, as handed down to us mortals by Agile Testing Experts and other assorted shills, is hocus. What? So it works in some situations, but not all. Thus hocus? I'm no 'Agile Testing Expert', but I have worked for the last 2-3 years doing TDD exclusively (as in ALWAYS writing tests first). It isn't hocus. It is effective. I can unders…

> I can understand it doesn't suit all problems, and can potentially be a hinderance sometimes Then why do it exclusively? Testing is about helping developers, and recognizing that automated testing is about benefit to developers, rather than cargo-culting a workflow and decreeing that one size fits all. Writing tests first as a tool to be deployed where it works is "Developer Driven Testing" - focussing on making th…

Primarily I do web development, with python and ruby, TDDing is VERY well supported, and thus has allowed me to do it exclusively. But I can imagine there are systems out there that are difficult to TDD, web apps are not among them though.

Sounds like you are interpreting TDD in a rather extreme way. If you haven't found it useful, then fine don't use it. But that doesn't mean it is universally bad, and thus doesn't deserve the "It's all hocus and you are a stupid zealot if you do it"

Re: Where unit testing fails

#48

Post by wnight »

peteretep wrote: post #11
Volpe wrote: post #8

Earlier quoted context omitted.

> The whole concept "test driven development" is hocus... I've worked on numerous projects where TDD has (and continues) to prove invaluable. So regardless how much experience you've got, I've had experience to the contrary, thus the "whole concept" isn't hocus. Why have you come to believe this?

It's hocus where contrasted to developer-driven testing, because Test Driven Development - as a development methodology (as opposed to a tool) - espouses that you Write All Your Tests first. Have I had experience (and much value) out of sometimes writing tests for certain problem classes before writing any code? Yes. Changes to existing functionality are often a good candidate. Does TDD as a methodology suggest you s…

> And "as any fule kno" [sic], this is idioicy during a design or hacking or greenfield phase of development.

When you write something you run it to make sure it works. Bottle that and call it a test. And then merely do this before you write the code. "What will I be done when I get this code to do?"

Don't plan, and spec, your whole API before you've written a line of code or anything, but do write a one-liner to check whatever it is you're going to do so you know when you're done.

Ideally you'll write a test (still just a one-liner) for every corner-case that you cover in code, because you need a failing test to know if your fix actually worked.

And face it, if you're doing "real" development work you've got business goals, and a spec with features intended to deliver on that, and you need to cover every tiny detail. If you don't have the spec ('fail on values over n') directly linked to the code that implements it, it will get lost, and if you aren't testing it you can't legitimately claim to be delivering it.

A spec, such as you'd get by performing useful TDD, is a requirement for modern software development. You could write it all up-front in the planning phase (waterfall style), or all after in the testing phase (waterfall style again), or you could interleave it with research and development, which if you think about it, sounds kind of agile by comparison...

Re: Where unit testing fails

#49

Post by wnight »

jsdalton wrote: post #21
peteretep wrote: post #11

Earlier quoted context omitted.

It's hocus where contrasted to developer-driven testing, because Test Driven Development - as a development methodology (as opposed to a tool) - espouses that you Write All Your Tests first. Have I had experience (and much value) out of sometimes writing tests for certain problem classes before writing any code? Yes. Changes to existing functionality are often a good candidate. Does TDD as a methodology suggest you s…

I've always thought of testing as simply a way to verify that your code does what is supposed to do. I tend to agree with you that TDD as a development methodology is deeply flawed. I have never found that the act of writing a test case and making it pass has helped me with the actual creative process of solving a problem via code. In fact, I'll frequently dash out a method I'm writing quickly, and then go back and r…

Perhaps by permuting your code the system could determine if there was anything that wasn't required to make the test pass and exclude it from the check-in.

I often code three or four commits worth at once when on a roll and it's a pain doing multiple staged adds, stashes, tests, and commits to store each sub-change on its own. Having a tool to help with this would be great.

Re: Where unit testing fails

#50

Post by wnight »

paganel wrote: post #22

Maybe I'm in a mad mood or somth (it's Monday, after all) but all this "let's write a parallel world, called TDD, to which the actual production code has to comply and praise and give sacrifices"-mantra has gotten a little over the top. Don't get me wrong, I'm sure that all these guys and chicks that live and die by TDD are smart (probably smarter than me), but what they are building is starting to get more and more…

Surely he did though. At some point he (roughly) had to add up the weight of the building and make sure the supports were thick enough to bear it. It's just too basic to talk about.
Post reply on HN