Where unit testing fails

I've always thought of testing as simply a way to verify that your code does what is supposed to do.

I tend to agree with you that TDD as a development methodology is deeply flawed. I have never found that the act of writing a test case and making it pass has helped me with the actual creative process of solving a problem via code. In fact, I'll frequently dash out a method I'm writing quickly, and then go back and reconstruct it piece by piece via TDD.

The main benefit writing tests first gives you is that you know your test is actually working (i.e. because it first fails) and that the code you implement actually passes it. It also keeps you honest because it prevents you from writing "extraneous" code, i.e. code that exists but is not specified by a test.

So yeah, I totally agree with you that the cult of TDD kind of sucks. But I haven't stumbled upon a better way yet. I've often thought it would be cool if there was some way for e.g. your version control software to be tied to the code you write that passes certain test(s), such that you could somehow verify that without that code your test fails and with that code your test passes. For the present though, TDD tends to work pretty well for me.

That's the point. Most unit tests wouldn't stop this kind of problem. ;)

I haven't had time to read them all yet so I was wondering if someone could give me a summary of why the TDD based sudoku approach failed?

Civil Engineers these days (or this is my understanding, anyway) DO use AutoCAD and 'all the fancy computer stuff' because testing allows you to code more productively, when you do it right. That's why engineers these days do automated simulations...

At the point where you're writing tests to impress other programmers (see article) about how much of a 'craftsman' you are, you lose. When you're writing tests that help you as a developer, and help get the software out more quickly and with higher quality, then you're doing it right.

I think there's 2 problem.

First, people rely too heavily on mocks/stubs/fakes/(whatever you want to call them). This has gotten better in the past 4 or so years (in my mind, largely because of Rails and it being "acceptable" to hit a DB in a test - though it might actually predate Rails). I think this problem is pretty straightforward to understand (again, especially when you look at the Rails way to test a DB interaction versus a more "traditional" way).

However, there are instances where mocks/fakes/stubs are important. Some outside dependencies might not be accessible during testing, might not be predictable/deterministic, might be too slow or might simply involve too much setup. Also, solely relying on "real" implementations can also make your tests too brittle. Why should the logic for can_legally_drink? break if the DB column is changed from DOB to dateofbirth (ok, that's an extreme and poor example, but you get the idea, hopefully).

Anyways, assuming you agree that sometimes a fake is simply better, you get into mocks vs stubs. To me a stub is dumb and a mock is strict. Stubs also automatically reply with canned answers, mocks don't. A mock is used to assert that a certain expected call was made. A stub is used just to get your test to move over a line of code.

The problem is that people use mocks over and over again. Re-specifying the same expectations in every test..making X tests break when you change the behavior of the interaction, versus just 1 test. As semi-pseudocode (and yes, it might be better to just hit the DB in this case):

  def self.login_user(name, password)
    user = Store.get_user_by_name(name)
    return user.nil? || user.password != password ? nil : user
  end

To me, you kinda wanna check that the above code properly interacts with someStore. This is when you use a mock:

  it "gets the user from the store" do
    Store.should_receive(:get_user_by_name).with('leto')
    User.login_user ('leto', 'ghanima')
  end

You also want to make sure the password matching works. This is where people go wrong. They'll use a mock (strict) again..which'll just repeat the above code...except this test has really nothing to do with how store...we just want to get over the line of code:

  it "returns nil if the passwords don't match" do
    Store.stub!(:get_user_by_name).and_return(User.new)
    User.login_user('leto', 'ghanima').should be_nil
  end

If you are familiar with jMock, it's kinda the difference between allowing and oneOf. oneOf is very strict, allowing isn't. You really should use allowing whenever you aren't explicitly testing the interaction (and you shouldn't explicitly test the interaction in more than 1 test (dedicated to testing said interaction)).

Some framework even return smart canned answers. So instead of returning nil, they'll default to a new instance of the type, or an empty array, or some default scalar value (like an empty string). This is particularly useful when you just want to get passed a null reference (which is exceedingly common).

Does that make sense?