Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Even Tetris is hard to test

blog.jwhitham.org

Post reply on HN
Print view View original

21–30 of 42 posts

Re: Even Tetris is hard to test

#21

Post by sargun »

gamegoblin wrote: post #3
forrestthewoods wrote: post #2

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

This is why I like using a randomized framework such as QuickCheck (or its derivatives in non-Haskell languages, such as Java) along with typical unit testing. It's often easier to write than tedious unit tests, and it can catch a lot of funny corner cases you don't even think to test. Of course, this still isn't a guarantee. But it does make me feel better about my code.

I love QuickCheck. It might be a little hard to do for Tetris.

Re: Even Tetris is hard to test

#22

Post by BrandonM »

aikah wrote: post #8
userbinator wrote: post #7

One way to reduce the number of possible paths is to reduce the number of distinct cases, often via refactoring or a simpler yet more general algorithm; to use an extremely contrived example, it's like the difference between int addOne(int x) { if(x == 0) return 1; else if(x == 1) return 2; ... } and int addOne(int x) { return x + 1; } I always keep in mind the famous Dijkstra quotes about testing and program complex…

Sorry but your exemple makes very little sense in the real world. Because what you wrote is obvious. Imagine now you have a switch with 50 different conditions that have nothing to do with each other and cant be reduced to a simple arithmetic operation.You'd have to test all the paths if you want a high test coverage rate. All you can do is abstract decision making through FP or OOP(chain of responsability).

I disagree that FP and OOP are the only abstraction methods. Arithmetic can be an excellent abstraction method.

For me, it's a code smell to have a function called "updateScore4". It's often possible to come up with an algebraic statement that gives the same result as a bunch of logic (code paths).

I think userbinator's point is that better code is easier to test as a result of having fewer code paths. Of course there are some gotchas to be aware of (e.g., overflow), but overall I agree.

Re: Even Tetris is hard to test

#23

Post by Someone »

canadev wrote: post #6
lucaspiller wrote: post #4

Earlier quoted context omitted.

One of the things I've found as a side effect of people using code coverage tools is that instead of testing the behaviour of a method they end up testing the implementation. I think this is because they initially test the behaviour, but then see that one path is missing, so add a test to ensure that path is ran - instead of just testing the behaviour that calls that path and checking the code coverage tool. This end…

I've faced this problem in my own tests. I want to achieve total coverage so that I know that I've got all the cases covered, but then I end up testing the implementation rather than the contract. I'm not sure what to do about it.

You cannot fully avoid that, but you can keep things organized. Do it Eiffel-ish.

At the start of your function, check all the prerequisites, e.g:

   if(x=n)throw "x should be smaller than n. Got $x and $n"
Add tests for edge conditions that do not throw, but, say, increase a global counting edge conditions hit:

  if(x=0) edgeConditionsHit += 1
  if(x=n) edgeConditionsHit += 1
Then, write tests so that you hit all paths in the condition tests.

If that doesn't hit 100% in the rest of the function, the function has code it doesn't need, or your precondition checks aren't complete.

Think about other edge conditions. For example, does your code special-case x=n/2? Add a check on top. And yes, that is implementation-specific, but there is nothing you can do about that.

Of course, you don't need the edge condition and implementation-specific checks in release builds.

With these in hand, you can also split tests into implementation-specific ones and contract-based ones.

Re: Even Tetris is hard to test

#24

Post by marcosdumay »

ygra wrote: post #17
forrestthewoods wrote: post #2

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

»Our customers tend to be makers of aircraft or car parts. Both businesses have strict safety standards which involve coverage testing, and our tools help you produce the relevant reports for certification, like DO-178B for the aviation industry.« I guess in both industries the value of more testing cannot be understated.

I guess you meant overstated...

And yes, it can. Are you trying to state their value compared to what? A good schema for task division, with encapsulation and whatchdogs? Proofs of correctness? Proofs of halting? Testing is much less valuable than any of those.

Re: Even Tetris is hard to test

#25

Post by gamegoblin »

sargun wrote: post #21
gamegoblin wrote: post #3

Earlier quoted context omitted.

This is why I like using a randomized framework such as QuickCheck (or its derivatives in non-Haskell languages, such as Java) along with typical unit testing. It's often easier to write than tedious unit tests, and it can catch a lot of funny corner cases you don't even think to test. Of course, this still isn't a guarantee. But it does make me feel better about my code.

I love QuickCheck. It might be a little hard to do for Tetris.

Definitely true, but with enough abstraction to small, composable, pure functions, you can get pretty far.

Re: Even Tetris is hard to test

#27

Post by nawitus »

BrandonM wrote: post #22
aikah wrote: post #8

Earlier quoted context omitted.

Sorry but your exemple makes very little sense in the real world. Because what you wrote is obvious. Imagine now you have a switch with 50 different conditions that have nothing to do with each other and cant be reduced to a simple arithmetic operation.You'd have to test all the paths if you want a high test coverage rate. All you can do is abstract decision making through FP or OOP(chain of responsability).

I disagree that FP and OOP are the only abstraction methods. Arithmetic can be an excellent abstraction method. For me, it's a code smell to have a function called "updateScore4". It's often possible to come up with an algebraic statement that gives the same result as a bunch of logic (code paths). I think userbinator's point is that better code is easier to test as a result of having fewer code paths. Of course ther…

On the other hand, you can "hide" the different code paths in a single algebraic statement, but even if your tests always execute that statement, you've lost information that you've actually tested the both "paths" of the algebraic statement.

I guess it's easier to use a boolean logic example (in JavaScript):

return myNormalObject || myDefaultObject();

Your code might always execute the myNormalObject half and return it, and even though your code coverage is 100% of the lines, your tests might miss myDefaultObject() code path.

Perhaps some code coverage tools can take this into consideration, though, but then you're back to the original problem..

Re: Even Tetris is hard to test

#28

Post by zach »

This is a great example of the value of test plans. This is basically technology to reconstruct missing test plans via the code. But of course someone already knew about the subtle "wall kick" feature since she or he wrote that code. It shouldn't be this hard, with some effective communication.

And actually, especially in games, test plans are still poorly communicated. In the old days, it was awful -- you would have the publisher doing all the QA, and barely speaking to the development team apart from bug reports. QA still often doesn't get involved until the last half of the project, before which time nobody has been thinking much about testing.

As studios improve their production, this is getting better. As a programmer, I've had more collaboration with QA as I work, at its best including having our QA liaison talk out a test plan with me while I'm working the feature. With enough communication, hopefully this kind of detective work to figure out what to test can be avoided.

Re: Even Tetris is hard to test

#29

Post by clebio »

mgraczyk wrote: post #9

Nitpick: As an avid Tetris player, I have to say that the examples given for "extremely rare" events actually happen quite often during normal play. In particular, clearing 4 lines at once, two times in row will generally happen within the first few minutes of game play and is sort of the whole goal (if you're trying to maximize points). Wall kicks are less common but certainly not rare enough that their processing c…

Are there more rare scenarios that you can think of?
Post reply on HN