Live data from Hacker News

Testing like the TSA

37signals.com

91–100 of 105 posts

Re: Testing like the TSA

#91
post #44

This is why I left Microsoft. Automated testing was a separate discipline - meaning there were "dev devs" and "test devs". Automated tests were written based on what the "test devs" had time for, not on the need or usefulness of such tests for the actual code. I was hired as a "test dev" - I had no industry experience at the time and figured I would give it an unprejudiced try to see if I liked it. I quickly realized…

While I agree with this description of the value of information, I disagree with your interpretation of it in this context. Consider the following, rather extreme example: nuclear power stations are equipped with countless diagnostic systems with several levels of fallback. In a well-built and well-operated nuclear power station these systems will never signal failure during normal operation. This clearly doesn't mea…

You are not designing nuclear power stations. You are at best designing a piece of business software.

(If you are designing a life-critical application, the scope of this article doesn't apply to you, as is stated in the article)

Re: Testing like the TSA

#93
post #66

Earlier quoted context omitted.

My own observation is that some ruby/rails developers get so enthralled with testing that it becomes an obsession that eclipses the product that they are building. The result (strong opinion coming...) is libraries like rspec and cucumber. They're complex and burdensome, and tend to attempt to mimic english, but often do so poorly and are totally unsuitable for non-coders. You spend a lot of time learning "the way" t…

For a long time, I had a deep hatred for RSpec because we used it on a project before the API stabilized (and before it was easy to maintain an environment where all developers had the same gems). We got stuck on some particular revision in the RSpec Subversion repository. The choice was re-write all the specs, or stick with that ancient version. We re-wrote all the specs -- to test/unit. Several years later, and I h…

There's a few solid reasons I've heard for using Cucumber, although in my life I've not found a need for it yet.

1) Although it's a Ruby tool, it works with a ton of languages. Someone can write code based off Cuke tests in Ruby or .NET or whatever without much trouble.

2) it makes web workflow testing cake

3) it keeps people strongly out of the "implementation" zone when they are thinking about how a program should be properly executed

4) it works with many spoken languages so if you're collaborating with an international team it could be useful there.

5) it has a whole bunch of report formats built in

If you don't find any of those features incredibly useful, I'm not sure you're going to ever see a need for it. I've played with it but for me it seems more hassle than anything. I do LIKE it but that's not enough to justify the time spent messing with it.

Re: Testing like the TSA

#95
This reminds me of the old joke about the man who went to his doctor and said: "Sir, those deposiwhatever pills you gave me were no good. I might just as well have stuck them up my ass".

Using a tool (or a medicine) to do something else than what it was designed for rarely has a positive effect. Cucumber was not designed to be a testing tool. Most people don't realise this, and end up with a verbose, unmaintainable mess. They blame the mess on the tool without realising that they used the tool wrong.

I don't use Cucumber to test my code. I use it to discover what code I need to write. This is a subtle, but important difference. This approach can work whether there are non-technical people involved or not.

I typically start out with a single Cucumber scenario that describes from 10.000 feet how I want a certain feature to work - without getting bogged down in details.

This allows me to reason and think about the domain in a way that helps me write the simplest possible code. What usually happens (to me at least) is that I end up with a simple design that reflects the domain.

Last week I wrote a small application like this. It is 1000 lines of Java code. I have 6 Cucumber scenarios - a total of 100 lines of Cucumber "code" (Gherkin) and about 200 lines of Step Definitions code (the glue code that sits between the Cucumber scenarios and the app).

I could have used JUnit instead - or I could have just written the code without any tests at all. For me, this would have made it harder to start coding. I would have started off with a much more muddy picture about how the app needs to behave and how it should be designed internally. I would have spent more time experimenting.

If you use Cucumber as a starting point to discover the code you need to write - and keep your scenarios few and high level - then you're more likely to reap benefits instead of pain.

And never fall for the temptation to use Cucumber to test details. That's what unit testing tools are for.

Re: Testing like the TSA

#96
I disagree about coverage.

First, I've found tests to be an excellent way to debug, much better than trying to manually get the app into the state I want and then stepping into the debugger.

Second, having a good coverage lets you refactor with a lot more confidence. I often open some file to fix a bug/add a feature and notice some piece of code that is horrible. I can comfortably fix that code too and if the tests pass (including the one that failed before fixing the initial bug/adding the feature) I'm done.

Re: Testing like the TSA

#97

Brilliant article. Testing for testings sake is wrong. Testing for 100% coverage sake is wrong. Write just enough tests at the level where it catches most of your regressions. Drill down into unit tests for complex logic, because you can test that more extensively and much faster than an integration test. Then leave a case or two for an integration test to make sure things are hooked up right. Don't be afraid to unit…

Write just enough tests at the level where it catches most of your regressions...make sure you didn't make a stupid off-by-one mistake. One thing that I've seen in inexperienced coders (including myself in the past) is that they tend to think of every bug as a fluke one-off mistake in an otherwise mostly flawless and awesome record. New coders tend to want to just fix a bug, then pretend it didn't happen. This is exa…

It seems so obvious to me now, but I just realised that is something that I do; I focusing on understanding what is causing a bug and how to fix it, but not why the bug occurred in the first place and how to prevent myself making similar mistakes in the future!

Thank you so much for making me aware of this.

Re: Testing like the TSA

#98
There are too many managers that just want Metrics. for example: Bill created 100 tests this month, Sam has 95% automated test coverage.

These metrics don't mean anything.

Re: Testing like the TSA

#99

You’re probably doing it wrong if testing is taking more than 1/3 of your time. You’re definitely doing it wrong if it’s taking up more than half. "No generalization is wholly true - not even this one" said Oliver Wendell Holmes. What proportion of your time do you spend writing tests for a one-off bash script to fix some filenames? What proportion of your time do you spend if your writing the fly-by-wire code in a 7…

Agreed, I see testing code and running code in an organic relationship. It make no sense to divide the time spent writing one or the other.

Re: Testing like the TSA

#100

You’re probably doing it wrong if testing is taking more than 1/3 of your time. You’re definitely doing it wrong if it’s taking up more than half. "No generalization is wholly true - not even this one" said Oliver Wendell Holmes. What proportion of your time do you spend writing tests for a one-off bash script to fix some filenames? What proportion of your time do you spend if your writing the fly-by-wire code in a 7…

Agreed, I see testing code and running code in an organic relationship. It make no sense to divide the time spent writing one or the other.
Post reply on HN