Even Tetris is hard to test
blog.jwhitham.org
Even Tetris is hard to test
1–10 of 42 posts
Re: Even Tetris is hard to test
#2Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?
Re: Even Tetris is hard to test
#3Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?
This is why I like using a randomized framework such as QuickCheck (or its derivatives in non-Haskell languages, such as Java) along with typical unit testing. It's often easier to write than tedious unit tests, and it can catch a lot of funny corner cases you don't even think to test.
Of course, this still isn't a guarantee. But it does make me feel better about my code.
Re: Even Tetris is hard to test
#4Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?
One of the things I've found as a side effect of people using code coverage tools is that instead of testing the behaviour of a method they end up testing the implementation. I think this is because they initially test the behaviour, but then see that one path is missing, so add a test to ensure that path is ran - instead of just testing the behaviour that calls that path and checking the code coverage tool. This ends up causing trouble if you ever want to change the implementation as you end up having to throw away half the tests, which means a lot of effort you spent to get 100% code coverage is now gone.
Re: Even Tetris is hard to test
#5TL;DR: Use a coverage tool for tests.
Re: Even Tetris is hard to test
#6Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?
One of the things I've found as a side effect of people using code coverage tools is that instead of testing the behaviour of a method they end up testing the implementation. I think this is because they initially test the behaviour, but then see that one path is missing, so add a test to ensure that path is ran - instead of just testing the behaviour that calls that path and checking the code coverage tool. This end…
I've faced this problem in my own tests. I want to achieve total coverage so that I know that I've got all the cases covered, but then I end up testing the implementation rather than the contract. I'm not sure what to do about it.
Re: Even Tetris is hard to test
#7One way to reduce the number of possible paths is to reduce the number of distinct cases, often via refactoring or a simpler yet more general algorithm; to use an extremely contrived example, it's like the difference between
int addOne(int x) {
if(x == 0)
return 1;
else if(x == 1)
return 2;
...
}
int addOne(int x) {
return x + 1;
}
"Program testing can be used to show the presence of bugs, but never to show their absence!"
"Simplicity is prerequisite for reliability."
Re: Even Tetris is hard to test
#8One way to reduce the number of possible paths is to reduce the number of distinct cases, often via refactoring or a simpler yet more general algorithm; to use an extremely contrived example, it's like the difference between int addOne(int x) { if(x == 0) return 1; else if(x == 1) return 2; ... } and int addOne(int x) { return x + 1; } I always keep in mind the famous Dijkstra quotes about testing and program complex…
Sorry but your exemple makes very little sense in the real world. Because what you wrote is obvious.
Imagine now you have a switch with 50 different conditions that have nothing to do with each other and cant be reduced to a simple arithmetic operation.You'd have to test all the paths if you want a high test coverage rate.
All you can do is abstract decision making through FP or OOP(chain of responsability).
Re: Even Tetris is hard to test
#9Nitpick: As an avid Tetris player, I have to say that the examples given for "extremely rare" events actually happen quite often during normal play. In particular, clearing 4 lines at once, two times in row will generally happen within the first few minutes of game play and is sort of the whole goal (if you're trying to maximize points). Wall kicks are less common but certainly not rare enough that their processing code would be left uncovered after playing a few games.
Re: Even Tetris is hard to test
#10One way to reduce the number of possible paths is to reduce the number of distinct cases, often via refactoring or a simpler yet more general algorithm; to use an extremely contrived example, it's like the difference between int addOne(int x) { if(x == 0) return 1; else if(x == 1) return 2; ... } and int addOne(int x) { return x + 1; } I always keep in mind the famous Dijkstra quotes about testing and program complex…
This is a perfect example of why line/function coverage is a silly measurement. It doesn't take into account global state and function input. Take the second function above, you could get 100% line coverage in just one run. But the function does exactly the same thing as the one where you only got 50% coverage, already things smell fishy. You can also test the function with millions of different inputs, all of them giving the same 100% coverage, and it will work perfectly fine, until suddenly you try addOne(int.max) and it will fail.
What you want to be really sure is state coverage, or input range coverage assuming your functions are pure. Now testing every function from int.min-int.max might seem unrealistic but what you have to do then is constrain the possible range of input or divide into ranges with special cases that you can somehow group together. Say for example int.min, negative numbers, zero, positive numbers and int.max.
Also, just because you covered a line doesn't mean it's correct, the only thing you've really tested is that the program doesn't crash. For the test to be really useful you also need a correct result of the output, added by a human. You can't just randomize input to increase the coverage.