Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Testing like the TSA

37signals.com

Post reply on HN
Print view View original

81–90 of 105 posts

Re: Testing like the TSA

#81

Post by [deleted] »

adrianhoward wrote: post #26
viraptor wrote: post #7

I don't get the "don't aim for 100%" point. What are you aiming for then? 50%? What happens when you reach 50% and remove some code - do you remove enough tests to match the ratio? We may just phrase the same idea in 2 different ways, but I'd go with "don't force yourself to do 100% coverage if it's not that relevant" / "don't add a test for a simple getter if you have more important things to do". If you can do 100%…

I don't get the "don't aim for 100%" point For me the point of that one is that test coverage isn't the goal - good code is. I've seen folk disappearing down a rabbit hole focusing on getting that last 2% of branch coverage using some baroque mock object monkey patched into the system. Their focus was on test coverage. What people who focus on test coverage get is an evil complex test suite that's very brittle in the…

[deleted]

Re: Testing like the TSA

#82

Post by pyre »

adrianhoward wrote: post #26
viraptor wrote: post #7

I don't get the "don't aim for 100%" point. What are you aiming for then? 50%? What happens when you reach 50% and remove some code - do you remove enough tests to match the ratio? We may just phrase the same idea in 2 different ways, but I'd go with "don't force yourself to do 100% coverage if it's not that relevant" / "don't add a test for a simple getter if you have more important things to do". If you can do 100%…

I don't get the "don't aim for 100%" point For me the point of that one is that test coverage isn't the goal - good code is. I've seen folk disappearing down a rabbit hole focusing on getting that last 2% of branch coverage using some baroque mock object monkey patched into the system. Their focus was on test coverage. What people who focus on test coverage get is an evil complex test suite that's very brittle in the…

  > damn - I can't test that easily - this code sucks
Depends on what you're doing. If you're writing a report that relies on a bunch of complex datasets, then it's a pain in the ass to generate said datasets, and calculate the expected result, but the test itself is fairly simple.

Re: Testing like the TSA

#83

Post by rntrg »

gfodor wrote: post #52

Hey, while we're slaughtering sacred cows, lets kill mocking, endotesting, expectation based testing, and the whole nine yards. It's a horrible practice that causes you to write too many tests, too many assertions, and results in tests that stay green even if you delete entire files from your codebase .

I have found that heavy use of mocking tools couples tests tightly to the design, resulting in great difficulty when I need to redactor. To me, the ideal is that I can change my design as I see fit without difficulty. My tests specify what my application does - how it behaves, not how it is structured. In my experience, this entails minimizing the number of points where tests touch production code, i.e. keeping the test suite DRY.

I do use test doubles, including mocks, where things become difficult or slow to test (generally application boundaries). But keep in mind that every line of mocking code costs more than an equivalent line of the alternative.

Re: Testing like the TSA

#84

Post by xianshou »

ejames wrote: post #70
SoftwareMaven wrote: post #65

Earlier quoted context omitted.

Your comment reads like you can know when a test will fail in the future (how else can you know the difference between a test that "always passes" and a test that will fail in the future to identify a regression?). You may have a test that passes for ten years. When do you know it's OK to nuke the test? Based on your follow-up, it is clear that my reading was not what you intended.

You can't know, but you can guess, based on past experience or logic. The simplest way to estimate the future is to guess that it will be similar to the past. For example, if you personally tend to write off-by-one errors a lot, it's a good idea to write tests which check that. On the other hand, if you almost never write off-by-one errors, you can skip those tests. If test is cheap to write, easy to investigate, and…

Re: "The simplest way to estimate the future is to guess that it will be similar to the past."

When we say that the future will be similar to the past, for code, we really mean that the probability of certain events occurring in the future will be similar to their prior probability of occurring in the past.

In my hypothetical example of testing an invariant that is unlikely to fail but damaging if it does, it might be valuable to keep that test around for five years even if it never fails. Imagine that the expected frequency of failure was initially , and that the test hasn't failed after five years. If the expected frequency of failure, cost of failure, and gain from fixing a failure remain the same, we should keep the test even if it's never failed: the expected benefit is constant.

Not to say that we should test for every possible bug, but if something is important enough in the first place to test for it, and that doesn't change (as calculated by expected benefit minus expected cost of maintenance), we should keep the test whether or not it changes our behavior.

Thus, if we could estimate probabilities correctly, we really would know when it's OK to nuke a test.

Re: Testing like the TSA

#85

Post by eru »

Dn_Ab wrote: post #80
tel wrote: post #21

Earlier quoted context omitted.

Unit testing didn't fully make sense to me until I played around with quickcheck (and eventually theorem proving in Coq). Unit tests vanish nicely to theorems and (empirical) proofs if your code expresses a succinct API. This is one end of the testing continuum. I use this sort of stuff extensively when doing mathematical computing and statistics because there's usually a clear mathematical boundary. Once you're insi…

I view unit tests as a kind of proof by counter-example. You have a logical structure your program embodies. This structure is very hard to specify mathematically and prove deductively so you come up with key statements that must at least evaluate to true for this structure/theory. The tests are a bunch of counter-examples that should be false (test passed). If a random testing framework is available in your language…

For Haskell you can do even better than random testing: There's Lazy SmallCheck for exhaustive testing. (For some values of `better' and `exhaustive'.)

Re: Testing like the TSA

#86

Post by MBlume »

zmoazeni wrote: post #18
raganwald wrote: post #8

Earlier quoted context omitted.

Strongly agree, but also I think that the interesting thing is why this might be so and what it tells us about application architecture. The basic thing about tests is that once you have them passing, they represent statements about constraints on the program. In other words, they express your opinion of things that should not change. Unit tests are a bet that certain aspects of the implementation will not change. In…

After playing with a language with a great type system (Haskell, not Java/C++), I've become more wary and borderline uncertain about my Ruby code. My integration tests serve two purposes: 1) Runs the code in an repeatable and isolated environment to verify I didn't do anything stupid like misname a variable, or treat something nil as an object. 2) Validate my unique application logic. I don't think #2 goes away with…

Note that, at least in python (which I assume is pretty similar to Ruby in ecosystem), you can get a lot of mileage for #1 by doing static analysis, for example Pylint.

Re: Testing like the TSA

#87

Post by ejames »

Splines wrote: post #72
ejames wrote: post #44

This is why I left Microsoft. Automated testing was a separate discipline - meaning there were "dev devs" and "test devs". Automated tests were written based on what the "test devs" had time for, not on the need or usefulness of such tests for the actual code. I was hired as a "test dev" - I had no industry experience at the time and figured I would give it an unprejudiced try to see if I liked it. I quickly realized…

Hey Evan ;) > We all realize there is no benefit in testing if you ignore failures rather than acting to fix the bugs, but in much the same way that doing nothing when tests fail has no benefit, doing nothing when tests pass also has no benefit - so tests which always pass are just as useless as failing tests you ignore, as are tests which only turn up corner-case bugs that you would have been comfortable with shippi…

Hey Jonathan ;)

This is the danger of having a 'test org' separate from the 'dev org'. When writing tests is tied to writing production code, your progress in one is tied to progress in the other. If it's easy to write tests for a particular feature, then the developer stops writing tests and writes the feature once they're done with the easy tests. It's much easier to understand your coverage when you're actually working with and writing the code being covered, rather than working in some separate "test org" - you don't need to run a coverage tool, you just see what breaks if you comment out a chunk of code. If the answer is "nothing" then it wasn't covered!

At the end of the day, an automated test suite is a product for developers on your project in the same way that a car is a product for drivers. You will have a hard time making a car if nobody on your car-making team can drive, and you will have a hard time writing a test suite as a tool to develop Project Foo if nobody on your test-writing team develops Project Foo.

I now write a project where I handle both the code and the tests. In the same way that my production code is improved by writing tests, my tests are improved by writing production code. I know what is included in the test coverage in the same way that I know what I had for lunch today - because I was there and I remember what happened. Tools for checking coverage are an aid to my understanding; in a company with a separate test org, you don't know anything about coverage until you've run the tool.

Re: Testing like the TSA

#88

Post by PatrickTulskie »

typicalrunt wrote: post #54
damoncali wrote: post #43

Don’t use Cucumber... Thank God someone with a bullhorn finally said this. I was beginning to think I was alone in my hatred of Cucumber. (And my love of Test:Unit/Minitest.)

I'm in the same boat as you, but I don't tend to voice my opinions on Test::Unit because there is such a strong opinion for RSpec and Cucumber. IMHO, I tend to like my tools to be tried and true, and not do any fancy magic.

I used Test::Unit for a long time, but I found it to be a poor tool when doing integration tests. RSpec lets me do both within the same DSL quite easily. That's probably why a lot of people flock to RSpec. I still have old Test::Unit code sitting in my RSpec suite that I haven't moved over to RSpec's DSL.

That being said, to a developer writing tests and doing TDD, Cucumber is a speed bump that doesn't need to be there. RSpec is great without lumping more crap on top of it.

Re: Testing like the TSA

#89

Post by stcredzero »

astral303 wrote: post #31
stcredzero wrote: post #27

Earlier quoted context omitted.

Write just enough tests at the level where it catches most of your regressions...make sure you didn't make a stupid off-by-one mistake. One thing that I've seen in inexperienced coders (including myself in the past) is that they tend to think of every bug as a fluke one-off mistake in an otherwise mostly flawless and awesome record. New coders tend to want to just fix a bug, then pretend it didn't happen. This is exa…

I think you might be misreading what I wrote. While your points are correct, I was specifically referring to "off-by-one mistake", which is a common "silly" error (since many indices are zero-based, it's often easy to request one too many elements, or chop off the first item). Also the way you quoted me above, "make sure you didn't make a stupid off-by-one mistake" looks like it's talking about writing just enough te…

I did misread what you wrote, but I thought the point was a good one to make, so posted it anyhow. Not disagreeing, just adding.

Re: Testing like the TSA

#90

Post by DrJokepu »

ejames wrote: post #44

This is why I left Microsoft. Automated testing was a separate discipline - meaning there were "dev devs" and "test devs". Automated tests were written based on what the "test devs" had time for, not on the need or usefulness of such tests for the actual code. I was hired as a "test dev" - I had no industry experience at the time and figured I would give it an unprejudiced try to see if I liked it. I quickly realized…

While I agree with this description of the value of information, I disagree with your interpretation of it in this context. Consider the following, rather extreme example: nuclear power stations are equipped with countless diagnostic systems with several levels of fallback. In a well-built and well-operated nuclear power station these systems will never signal failure during normal operation. This clearly doesn't mean that the output of these systems carries little value. Surely, a test that is always passing doesn't necessarily has no benefit, you also have to consider what it would mean if it suddenly stopped passing.
Post reply on HN