Testing like the TSA • Hacker News Board

Re: Testing like the TSA

#51

Post by angersock » Wed, Apr 11, 2012, 7:23 PM UTC

jashkenas wrote: post #2

A thought that folks reading this post might have an opinion on: "Libraries should be mostly unit tested. Applications should be mostly (and lightly) integration tested. Naturally, some parts of a complex app will behave like a library..." Agree or disagree?

We recently finished approximately one metric fuckton of unit tests for the next release of some of our libraries, and I tend to agree very strongly that a library must be well-tested.

A problem that we ran into, though, is exactly what to test. The biggest basic distinction we ran into was testing to verify interface integrity (i.e., does this do what it says in the header documentation, pursuant to declared constraints and so on) as opposed to implementation integrity (does my ringbuffer properly move around elements, does my array resize move my elements, etc.).

The former is very useful to ensure that the library is good for users, the latter of course is much better for testing that changes to the containers don't do something stupid.

Re: Testing like the TSA

#52

Post by gfodor » Wed, Apr 11, 2012, 7:24 PM UTC

Hey, while we're slaughtering sacred cows, lets kill mocking, endotesting, expectation based testing, and the whole nine yards. It's a horrible practice that causes you to write too many tests, too many assertions, and results in tests that stay green even if you delete entire files from your codebase.

Re: Testing like the TSA

#53

Post by ejames » Wed, Apr 11, 2012, 7:29 PM UTC

SoftwareMaven wrote: post #47

ejames wrote: post #44

This is why I left Microsoft. Automated testing was a separate discipline - meaning there were "dev devs" and "test devs". Automated tests were written based on what the "test devs" had time for, not on the need or usefulness of such tests for the actual code. I was hired as a "test dev" - I had no industry experience at the time and figured I would give it an unprejudiced try to see if I liked it. I quickly realized…

I disagree that there is no benefit in passing tests that don't change your behavior. Those tests are markers to prevent you from unknowingly doing something that should have changed your behavior. That is where the nuance enters: is this a marker I want to lay down or not? Some markers should be there; others absolutely should not and just introduce noise.

I don't understand what you mean by "prevent you from unknowingly doing something that should have changed your behavior". If you do something without knowing it, how could it change your behavior? If it's a case where you should have changed your behavior, why would you prevent it?

Re: Testing like the TSA

#54

Post by typicalrunt » Wed, Apr 11, 2012, 7:43 PM UTC

damoncali wrote: post #43

Don’t use Cucumber... Thank God someone with a bullhorn finally said this. I was beginning to think I was alone in my hatred of Cucumber. (And my love of Test:Unit/Minitest.)

I'm in the same boat as you, but I don't tend to voice my opinions on Test::Unit because there is such a strong opinion for RSpec and Cucumber. IMHO, I tend to like my tools to be tried and true, and not do any fancy magic.

Re: Testing like the TSA

#55

Post by xianshou » Wed, Apr 11, 2012, 7:44 PM UTC

ejames wrote: post #53

SoftwareMaven wrote: post #47

Earlier quoted context omitted.

I disagree that there is no benefit in passing tests that don't change your behavior. Those tests are markers to prevent you from unknowingly doing something that should have changed your behavior. That is where the nuance enters: is this a marker I want to lay down or not? Some markers should be there; others absolutely should not and just introduce noise.

I don't understand what you mean by "prevent you from unknowingly doing something that should have changed your behavior". If you do something without knowing it, how could it change your behavior? If it's a case where you should have changed your behavior, why would you prevent it?

I believe the above comment refers to regression testing. For instance, if I write a test for an invariant that is fairly unlikely to change, then the chance that my behavior will change in the next hour based on the test run is small. However, if and when the invariant is mistakenly changed, even though negative side effects might not be immediately visible, it could be immensely valuable to me to see the flaw and restore that invariant.

Re: Testing like the TSA

#56

Post by matwood » Wed, Apr 11, 2012, 7:46 PM UTC

adrianhoward wrote: post #39

You’re probably doing it wrong if testing is taking more than 1/3 of your time. You’re definitely doing it wrong if it’s taking up more than half. "No generalization is wholly true - not even this one" said Oliver Wendell Holmes. What proportion of your time do you spend writing tests for a one-off bash script to fix some filenames? What proportion of your time do you spend if your writing the fly-by-wire code in a 7…

Also, to be honest, I'd be hard put to tell anybody how much time I spend writing tests vs. writing code.

That's a good point. Test writing versus code writing is not black and white. Spending time writing tests is also time spent thinking about what's going to be coded. Writing tests should make the coding process go quicker such that there exists a lot of overlap in test writing and coding.

Re: Testing like the TSA

#57

Post by xianshou » Wed, Apr 11, 2012, 7:51 PM UTC

skrebbel wrote: post #32

Nice read. I'm no Rails dev, so I'm curious about this one point from DHH: > 6. Don't use Cucumber unless you live in the magic kingdom of non-programmers-writing-tests (and send me a bottle of fairy dust if you're there!) I mostly do C#, and teams I've recently been on have found SpecFlow tests to be an excellent time saver in communicating requirements and acceptance test criteria with customers. Has Cucumber not b…

Personally, I don't like Cucumber because it attempts to bridge the worlds of development and business but ends up serving neither very well. Its syntax gives it the verbosity of English while removing none of the brittleness of the underlying code. A non-programmer editing tests would need to learn Ruby to deal with the leaks in the abstraction, while a coder is better off describing behavior directly in rspec or Capybara rather than wrapping it in a "story" that should have been a two-line comment.

As for non-programmers reading tests, they could read the comments or README.md instead. Why force the documentation to test, and the test to document, if it decreases the efficiency of both?

Re: Testing like the TSA

#58

Post by [deleted] » Wed, Apr 11, 2012, 7:52 PM UTC

ejames wrote: post #53

SoftwareMaven wrote: post #47

Earlier quoted context omitted.

I disagree that there is no benefit in passing tests that don't change your behavior. Those tests are markers to prevent you from unknowingly doing something that should have changed your behavior. That is where the nuance enters: is this a marker I want to lay down or not? Some markers should be there; others absolutely should not and just introduce noise.

I don't understand what you mean by "prevent you from unknowingly doing something that should have changed your behavior". If you do something without knowing it, how could it change your behavior? If it's a case where you should have changed your behavior, why would you prevent it?

[deleted]

Re: Testing like the TSA

#59

Post by BadassFractal » Wed, Apr 11, 2012, 8:02 PM UTC

ejames wrote: post #44

This is why I left Microsoft. Automated testing was a separate discipline - meaning there were "dev devs" and "test devs". Automated tests were written based on what the "test devs" had time for, not on the need or usefulness of such tests for the actual code. I was hired as a "test dev" - I had no industry experience at the time and figured I would give it an unprejudiced try to see if I liked it. I quickly realized…

I worked on enterprise projects "at a similar company" with a few hundred "top notch" engineers. We had no more than 30% coverage, most of it from SDETs. No tests were being written by the devs before checking in for months, the test devs were understaffed, unqualified and thus behind. At one point someone made a checkin that prevented login from happening. Nobody noticed for a full work week, until that version made it into preproduction and someone finally attempted to login. Apparently hundreds of mils spent on the project can't buy you a team able to write above high-school level code.

I can see the usefulness of SDETs in doing system / end-to-end testing or testing of really obscure scenarios, but most of the test writing should belong to devs. I love the Rails approach to UT, functional and integration test split. The first time you try BDD, especially if you're coming from an after-the-fact testing culture like the one above, you almost want to cry from joy. I agree that Cucumber might a bit of an overkill, but perhaps I don't get it. For a non-prototypey project you should absolutely add other types horizontal testing like performance, security..

Re: Testing like the TSA

#60

Post by ejames » Wed, Apr 11, 2012, 8:06 PM UTC

xianshou wrote: post #55

ejames wrote: post #53

Earlier quoted context omitted.

I don't understand what you mean by "prevent you from unknowingly doing something that should have changed your behavior". If you do something without knowing it, how could it change your behavior? If it's a case where you should have changed your behavior, why would you prevent it?

I believe the above comment refers to regression testing. For instance, if I write a test for an invariant that is fairly unlikely to change, then the chance that my behavior will change in the next hour based on the test run is small. However, if and when the invariant is mistakenly changed, even though negative side effects might not be immediately visible, it could be immensely valuable to me to see the flaw and r…

Yes - but the test would fail when the invariant is mistakenly changed. On the test run after the invariant was changed, you would get new information (the test does not always pass) and change your behavior (revert the commit which altered the invariant).

That is the point of the "changing behavior" rule - you do not gather the benefit of running a test until it has failed at least once, and the benefit gathered is proportionate to the benefit of the action you take upon seeing the failure. The tricky part of the rule is that you must predict your actions in the future, since a test that might have a very important failure later could pass all the time right now. Knowing your own weaknesses and strengths is important, as is knowing the risks of your project.

There are possible design benefits to writing tests, since you must write code that is testable, and testable code tends to also be modular. However, once you have written testable code, you still gain those design benefits even if you never run your test suite, or even delete your tests entirely!