Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing like the TSA

37signals.com

Post reply on HN
Print view View original

31–40 of 105 posts

Re: Testing like the TSA

#31

Post by astral303 »

stcredzero wrote: post #27
astral303 wrote: post #5

Brilliant article. Testing for testings sake is wrong. Testing for 100% coverage sake is wrong. Write just enough tests at the level where it catches most of your regressions. Drill down into unit tests for complex logic, because you can test that more extensively and much faster than an integration test. Then leave a case or two for an integration test to make sure things are hooked up right. Don't be afraid to unit…

Write just enough tests at the level where it catches most of your regressions...make sure you didn't make a stupid off-by-one mistake. One thing that I've seen in inexperienced coders (including myself in the past) is that they tend to think of every bug as a fluke one-off mistake in an otherwise mostly flawless and awesome record. New coders tend to want to just fix a bug, then pretend it didn't happen. This is exa…

I think you might be misreading what I wrote. While your points are correct, I was specifically referring to "off-by-one mistake", which is a common "silly" error (since many indices are zero-based, it's often easy to request one too many elements, or chop off the first item).

Also the way you quoted me above, "make sure you didn't make a stupid off-by-one mistake" looks like it's talking about writing just enough tests. However, in context, I'm actually referring to writing unit tests for small items where you might make a stupid off-by-one mistake.

So I was never referring to "one-off mistakes", as in mistakes that are flukes.

Your points are all good otherwise! Never rest on your laurels and always think about what you can do to catch your mistakes.

Re: Testing like the TSA

#32

Post by skrebbel »

Nice read.

I'm no Rails dev, so I'm curious about this one point from DHH:

> 6. Don't use Cucumber unless you live in the magic kingdom of non-programmers-writing-tests (and send me a bottle of fairy dust if you're there!)

I mostly do C#, and teams I've recently been on have found SpecFlow tests to be an excellent time saver in communicating requirements and acceptance test criteria with customers. Has Cucumber not been designed for the same purpose?

I might guess that David included the point because a product business such as 37signals has no non-programming stakeholders to communicate about requirements and acceptance criteria with.

Using BDD for having non-programmers write tests sounds far-fetched to me indeed. It's excellent to have them able to read and understand the tests, though. Any opinions? Is BDD as dead horse, or is DHH a little narrow minded here?

Re: Testing like the TSA

#33

Post by adrianhoward »

Code-to-test ratios above 1:2 is a smell, above 1:3 is a stink

I think this depends on the domain. Some code ends up with a metric shed load of important edge cases because it's modelling something with a metric shed load of important edge cases - not just because your code sucks.

For example one project I worked on involved a lot of code to manage building estimates. It involved a stack of special cases related to building codes, different environments, the heuristics that the human estimators used, etc.

There wasn't any sane way to remove the special cases - the domain caused them. There wasn't a way to sensibly avoid writing tests - since we didn't really want estimates for the amount of drywall in a multi-million pound skyscraper to be wrong :-)

Re: Testing like the TSA

#34

Post by hkarthik »

Discussions like this arise mostly because developers today think less about whether a feature should be built and more about how it should be architected and tested.

For decades, we've been conditioned to think that by the time a feature is written down on paper, someone else has crossed all the Ts and dotted all the Is to determine that the feature has value and should be built.

It's far easier to focus on low value testing issues than attempt the harder work of convincing the customer/product owner that certain features should not be built.

Re: Testing like the TSA

#35

Post by samd »

Testing isn't just about preventing bugs, it also encourages better design, and good design makes extending and enhancing your application much easier and faster.

...in general, of course, there's always exceptions, don't be dogmatic.

Re: Testing like the TSA

#36

Post by excuse-me »

Testing, or at least coding, like the TSA thinks is remarkably common in industry.

TSA logic - somebody tried to get on a plane with a bomb in their show so everyone everywhere has to remove their shoes

PHB logic - somebody somewhere had a bug because of a misuse of inheritance so you aren't allowed to use any inheritance in your C++ code anywhere.

Re: Testing like the TSA

#37

Post by bryanlarsen »

zmoazeni wrote: post #24

@dhh was talking about this on twitter before he made this post. And I think @dchelimsky has a very fair point that @dhh ignores: testing is all about effort to eliminate some amount of risk. Test x but don't test y can't be universal. Risks are (often radically) different in every app/team. (from https://twitter.com/#!/dchelimsky/status/190096365794230274 ) @dhh and 37Signals have a different perspective on acceptab…

I think dhh tried to address that with his comment yes, yes, if you were working on an airport control system for launching rockets to Mars and the rockets would hit the White House if they weren’t scheduled with a name, you can test it—but you aren’t, so forget it

Re: Testing like the TSA

#38

Post by zmoazeni »

bryanlarsen wrote: post #37
zmoazeni wrote: post #24

@dhh was talking about this on twitter before he made this post. And I think @dchelimsky has a very fair point that @dhh ignores: testing is all about effort to eliminate some amount of risk. Test x but don't test y can't be universal. Risks are (often radically) different in every app/team. (from https://twitter.com/#!/dchelimsky/status/190096365794230274 ) @dhh and 37Signals have a different perspective on acceptab…

I think dhh tried to address that with his comment yes, yes, if you were working on an airport control system for launching rockets to Mars and the rockets would hit the White House if they weren’t scheduled with a name, you can test it—but you aren’t, so forget it

That's a hyperbolic comparison. Yes, focusing on outliers makes all the other data points look the same. My point is that in the context of each team/app, those differences in acceptable risk matter.

Re: Testing like the TSA

#39

Post by adrianhoward »

You’re probably doing it wrong if testing is taking more than 1/3 of your time. You’re definitely doing it wrong if it’s taking up more than half.

"No generalization is wholly true - not even this one" said Oliver Wendell Holmes.

What proportion of your time do you spend writing tests for a one-off bash script to fix some filenames? What proportion of your time do you spend if your writing the fly-by-wire code in a 777? You should spend "enough" time writing tests - where "enough" is extremely context dependent.

I also worry about this sort of advice because the reason many spend too long testing is that they're really bad a testing. I fear this group will use the quote as an excuse to do less testing, rather than get better at testing.

Also, to be honest, I'd be hard put to tell anybody how much time I spend writing tests vs. writing code. I've been practicing TDD for about ten years now - and I just don't think about "testing" vs "coding". I'm developing - which involves writing tests and writing code. Trying to separate them out and time them makes about as much sense to me as worrying about whether I type more with my left or right hand.

Re: Testing like the TSA

#40

Post by chrisdinn »

jashkenas wrote: post #2

A thought that folks reading this post might have an opinion on: "Libraries should be mostly unit tested. Applications should be mostly (and lightly) integration tested. Naturally, some parts of a complex app will behave like a library..." Agree or disagree?

Agree, but I'm concerned all these rules of thumb are doing nothing to help the authors of bad test suites understand why they're bad and how to fix them.

In my opinion the point of building out a test suite isn't to reach 100% code coverage, it's to allow developers working on the application the ability to refactor quickly and aggressively during the development process while remaining confident that it will still perform correctly when they're ready to release. Poorly written tests (or over-testing) will slow down refactoring and hinder development. But, in my experience, not having the right tests can mean firing up your application and manually testing after each small code change to prevent regression. Understanding the trade-off is key to helping developers answer the ubiquitous question: "do I need to test this"?

This role for unit tests is described effectively in Chapter 1 of Martin Fowler's awesome Refactoring book (or Refactoring, Ruby Edition, if that suites you. That's the version that's on my desk right now.) This use case is also highlighted in Eric Evans' book Domain Driven Design, where he highlights one of my favourite application development techniques: "refactoring to deeper insight." Both books are among my favourites.

Adding features to an application isn't just about appending files and lines of code, it's also about changing existing code to allow the new features to fit comfortably into the application's domain model. If you can accomplish that goal with "light integration tests" go for it. That'll mean much less overhead during refactoring and will do a good job of enforcing correctness. If your application contains a robust domain model (which it very well might) you may find unit tests useful for maintaining model integrity during development. This is probably what's meant by "Naturally, some parts of a complex app will behave like a library". When you change the domain model you'll have to change tests for all the affected classes and that's a good thing. It explicitly highlights how your change affected the model, which you should certainly understand before committing your changes.

I will admit this distinction is hard to communicate to first-time TDDers. However, I've found that after having to eat their own dog food for a few iterations (adding features to your own code) smart developers quickly find the sweet spot. The less code you write, the less you have to maintain. But too few test or poorly written tests can slow you down just as much.

Post reply on HN