Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing like the TSA

37signals.com

Post reply on HN
Print view View original

61–70 of 105 posts

Re: Testing like the TSA

#61

Post by 100k »

I am glad DHH has put up this piece. This is one of my favorite topics to be contrarian about. I am a strong believer in the importance of developer testing to ensure code quality, but some people take it way too far. I gave a talk about this at RubyFringe: http://railspikes.com/2008/7/11/testing-is-overrated

I also am utterly opposed to the Cucumber-style programming-in-English-via-regexp testing approach. Unless someone who doesn't know how to code is writing those step files, why subject yourself to that?!

Re: Testing like the TSA

#62

Post by 100k »

gfodor wrote: post #52

Hey, while we're slaughtering sacred cows, lets kill mocking, endotesting, expectation based testing, and the whole nine yards. It's a horrible practice that causes you to write too many tests, too many assertions, and results in tests that stay green even if you delete entire files from your codebase .

I would like to keep an open mind about this because I like the theory of decoupling.

But the only thing I use mocking/stubbing for is to mock out external services that make tests unreliable and slow (like HTTP requests).

Expectation based tests seem particularly easy to mis-use. Who cares how many times a method is called?

Re: Testing like the TSA

#63

Post by pyrhho »

viraptor wrote: post #7

I don't get the "don't aim for 100%" point. What are you aiming for then? 50%? What happens when you reach 50% and remove some code - do you remove enough tests to match the ratio? We may just phrase the same idea in 2 different ways, but I'd go with "don't force yourself to do 100% coverage if it's not that relevant" / "don't add a test for a simple getter if you have more important things to do". If you can do 100%…

As a counter-argument to the "don't aim for 100%" thing, I would say that code coverage is more important in interpreted (and loosely typed) languages. Particularly with preventing regressions. For example, if you change the arity of a function, it's nice to be able to run the tests and see all the calls to it that break.

Re: Testing like the TSA

#64

Post by stonemetal »

jashkenas wrote: post #2

A thought that folks reading this post might have an opinion on: "Libraries should be mostly unit tested. Applications should be mostly (and lightly) integration tested. Naturally, some parts of a complex app will behave like a library..." Agree or disagree?

I am not sure I understand the point of making code into a library if it isn't going to be reused outside of the application. YAGNI and all that.

I would agree with unit test logic, integration test glue, but that is heading into truism country.

Re: Testing like the TSA

#65

Post by SoftwareMaven »

ejames wrote: post #60
xianshou wrote: post #55

Earlier quoted context omitted.

I believe the above comment refers to regression testing. For instance, if I write a test for an invariant that is fairly unlikely to change, then the chance that my behavior will change in the next hour based on the test run is small. However, if and when the invariant is mistakenly changed, even though negative side effects might not be immediately visible, it could be immensely valuable to me to see the flaw and r…

Yes - but the test would fail when the invariant is mistakenly changed. On the test run after the invariant was changed, you would get new information (the test does not always pass) and change your behavior (revert the commit which altered the invariant). That is the point of the "changing behavior" rule - you do not gather the benefit of running a test until it has failed at least once, and the benefit gathered is…

Your comment reads like you can know when a test will fail in the future (how else can you know the difference between a test that "always passes" and a test that will fail in the future to identify a regression?). You may have a test that passes for ten years. When do you know it's OK to nuke the test?

Based on your follow-up, it is clear that my reading was not what you intended.

Re: Testing like the TSA

#66

Post by 100k »

damoncali wrote: post #42
skrebbel wrote: post #32

Nice read. I'm no Rails dev, so I'm curious about this one point from DHH: > 6. Don't use Cucumber unless you live in the magic kingdom of non-programmers-writing-tests (and send me a bottle of fairy dust if you're there!) I mostly do C#, and teams I've recently been on have found SpecFlow tests to be an excellent time saver in communicating requirements and acceptance test criteria with customers. Has Cucumber not b…

My own observation is that some ruby/rails developers get so enthralled with testing that it becomes an obsession that eclipses the product that they are building. The result (strong opinion coming...) is libraries like rspec and cucumber. They're complex and burdensome, and tend to attempt to mimic english, but often do so poorly and are totally unsuitable for non-coders. You spend a lot of time learning "the way" t…

For a long time, I had a deep hatred for RSpec because we used it on a project before the API stabilized (and before it was easy to maintain an environment where all developers had the same gems).

We got stuck on some particular revision in the RSpec Subversion repository. The choice was re-write all the specs, or stick with that ancient version. We re-wrote all the specs -- to test/unit.

Several years later, and I have never picked up RSpec for my own use. However, I am working on another project that chose RSpec and it is working out pretty well. I have turned on render_views so I don't have to test those separately and am only using mocking for external services.

Cucumber, on the other hand, I do not understand at all. Why write tests in English when you have Ruby?

Re: Testing like the TSA

#67

Post by namidark »

What a bunch of hogwash; 1000 lines of code to to test validates_presence_of? What testing framework is he using? x86 assembler?

The only part I agree about is the view testing and partially cucumber.

Then again I dont think 37s has a large amount of business logic to test (compared to some of the other commenters in here) and their testing is mostly front end, and testing front end ajax-y stuff with cucumber, does indeed suck.

Re: Testing like the TSA

#68

Post by aneth »

gfodor wrote: post #52

Hey, while we're slaughtering sacred cows, lets kill mocking, endotesting, expectation based testing, and the whole nine yards. It's a horrible practice that causes you to write too many tests, too many assertions, and results in tests that stay green even if you delete entire files from your codebase .

How would you test, say, payment processing, in any sane way, without mocking?

How do you test failure handling code when the failure can not be reproduced in a known way by the current code base?

I agree these tools are often overused, however they do have their roles.

Re: Testing like the TSA

#69

Post by equalarrow »

typicalrunt wrote: post #54
damoncali wrote: post #43

Don’t use Cucumber... Thank God someone with a bullhorn finally said this. I was beginning to think I was alone in my hatred of Cucumber. (And my love of Test:Unit/Minitest.)

I'm in the same boat as you, but I don't tend to voice my opinions on Test::Unit because there is such a strong opinion for RSpec and Cucumber. IMHO, I tend to like my tools to be tried and true, and not do any fancy magic.

Learned my lesson the hard way(s) with Rspec and Cucumber a while ago - total waste of my time.

There's a group of devs that have fooled themselves into thinking anyone outside their group understands how they are testing. I've seen this firsthand. DHH has always been right, imo, in this regard; test what you think is important, use simple tools.

TestUnit still serves me well and it perfectly fits my needs of do more with less.

Re: Testing like the TSA

#70

Post by ejames »

SoftwareMaven wrote: post #65
ejames wrote: post #60

Earlier quoted context omitted.

Yes - but the test would fail when the invariant is mistakenly changed. On the test run after the invariant was changed, you would get new information (the test does not always pass) and change your behavior (revert the commit which altered the invariant). That is the point of the "changing behavior" rule - you do not gather the benefit of running a test until it has failed at least once, and the benefit gathered is…

Your comment reads like you can know when a test will fail in the future (how else can you know the difference between a test that "always passes" and a test that will fail in the future to identify a regression?). You may have a test that passes for ten years. When do you know it's OK to nuke the test? Based on your follow-up, it is clear that my reading was not what you intended.

You can't know, but you can guess, based on past experience or logic. The simplest way to estimate the future is to guess that it will be similar to the past.

For example, if you personally tend to write off-by-one errors a lot, it's a good idea to write tests which check that. On the other hand, if you almost never write off-by-one errors, you can skip those tests. If test is cheap to write, easy to investigate, and covers a piece of code that would cause catastrophic problems if it failed, it's worthwhile to write the test even if you can barely imagine a possible situation where it would fail - the degree of the cost matters as much as the degree of the benefit.

You don't "know" when it's OK to nuke a test just as you don't really "know" when it's safe to launch a product - you decide what you're doing based on experience, knowledge, and logic. The important step many don't take is developing the ability to distinguish between good tests and bad tests, rather than simply having an opinion on testing in general.

Post reply on HN