Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Even Tetris is hard to test

blog.jwhitham.org

Post reply on HN
Print view View original

31–40 of 42 posts

Re: Even Tetris is hard to test

#31

Post by GFK_of_xmaspast »

lucaspiller wrote: post #4
forrestthewoods wrote: post #2

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

One of the things I've found as a side effect of people using code coverage tools is that instead of testing the behaviour of a method they end up testing the implementation. I think this is because they initially test the behaviour, but then see that one path is missing, so add a test to ensure that path is ran - instead of just testing the behaviour that calls that path and checking the code coverage tool. This end…

If you write a test that forces a particular codepath in the original implementation, like "parameter one is an empty string and parameter two is a null pointer, and verify the return value of the function call", then it should still be a perfectly valid test if the implementation changes, it just might not be a very meaningful test.

Re: Even Tetris is hard to test

#32

Post by shalmanese »

clebio wrote: post #29
mgraczyk wrote: post #9

Nitpick: As an avid Tetris player, I have to say that the examples given for "extremely rare" events actually happen quite often during normal play. In particular, clearing 4 lines at once, two times in row will generally happen within the first few minutes of game play and is sort of the whole goal (if you're trying to maximize points). Wall kicks are less common but certainly not rare enough that their processing c…

Are there more rare scenarios that you can think of?

There's a lot of special conditions around having almost the entire playing field filled in.

Re: Even Tetris is hard to test

#33

Post by asveikau »

Too wrote: post #10
userbinator wrote: post #7

One way to reduce the number of possible paths is to reduce the number of distinct cases, often via refactoring or a simpler yet more general algorithm; to use an extremely contrived example, it's like the difference between int addOne(int x) { if(x == 0) return 1; else if(x == 1) return 2; ... } and int addOne(int x) { return x + 1; } I always keep in mind the famous Dijkstra quotes about testing and program complex…

This is a perfect example of why line/function coverage is a silly measurement. It doesn't take into account global state and function input. Take the second function above, you could get 100% line coverage in just one run. But the function does exactly the same thing as the one where you only got 50% coverage, already things smell fishy. You can also test the function with millions of different inputs, all of them g…

I came to this thread hoping somebody would say pretty much exactly your first paragraph. The author appears to be selling a code coverage tool so of course he falls into the trap. In real life you have to remember that hitting a line once is not the same as showing it to be correct. People who buy into code coverage tools make this mistake a lot.

Re: Even Tetris is hard to test

#35

Post by Dewie »

BrandonM wrote: post #22
aikah wrote: post #8

Earlier quoted context omitted.

Sorry but your exemple makes very little sense in the real world. Because what you wrote is obvious. Imagine now you have a switch with 50 different conditions that have nothing to do with each other and cant be reduced to a simple arithmetic operation.You'd have to test all the paths if you want a high test coverage rate. All you can do is abstract decision making through FP or OOP(chain of responsability).

I disagree that FP and OOP are the only abstraction methods. Arithmetic can be an excellent abstraction method. For me, it's a code smell to have a function called "updateScore4". It's often possible to come up with an algebraic statement that gives the same result as a bunch of logic (code paths). I think userbinator's point is that better code is easier to test as a result of having fewer code paths. Of course ther…

> I disagree that FP and OOP are the only abstraction methods. Arithmetic can be an excellent abstraction method.

Arithmetic in programming is typically about functions taking in numbers and returning new numbers, ie. returning a new number instead of mutating one of the numbers in-place. That sounds like the spirit of FP, to me. (Though of course arithmetic on fixed-size numbers falls short of this ideal when it comes to overflow and such, and we often don't handle this possibility.)

I don't doubt that there are other abstractions than FP and OOP. But arithmetic looks like FP, to me.

Re: Even Tetris is hard to test

#36

Post by dack »

forrestthewoods wrote: post #2

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

One thing I like to say when people bring up test coverage is that code being "covered" only says it was run, not that it was correct, so it's only a weak statement of quality. However, code that is not covered is completely unknown, so there is a bigger chance of bugs there. Obviously though, if that code is trivial, it may not be worth the maintenance overhead of another test for just that.

So yeah, when a team claims 100% code coverage, usually that is just a signal that they care about testing and the quality of the code, therefore it tends to be less buggy. Not necessarily because 100% coverage itself made it so.

Really I only use a code coverage tool to check for important places that aren't covered at all, AFTER I have tried to think of the proper behavior/spec of the code from an outside perspective. It's like a secondary check after you think you are already done. That keeps you focused on what correct input and output are, and then patching up the little areas that you missed with a tool.

Re: Even Tetris is hard to test

#37

Post by fidotron »

This is amusingly naive. I've seen the spec for Tetris, and it's surprisingly big - much larger than the smallest known fully conforming code, which is still surprisingly large. There are also a non tiny number of people knocking around for whom it's their entire job to test it. "Even" Tetris? No, sorry - that's nonsense.

The big problem here is that code coverage tests don't help you cover what you should have explicitly defined or tested but didn't. As a result a lot of things end up still defined by implementation and not specification, as all sorts of important details only got defined during implementation.

Re: Even Tetris is hard to test

#38

Post by nanomage »

zach wrote: post #28

This is a great example of the value of test plans. This is basically technology to reconstruct missing test plans via the code. But of course someone already knew about the subtle "wall kick" feature since she or he wrote that code. It shouldn't be this hard, with some effective communication. And actually, especially in games, test plans are still poorly communicated. In the old days, it was awful -- you would have…

Agreed. Dev's see the specs, and QA sees the holes. I get to participate from the initial planning into release. As a QA person I feel very lucky.

In regards to the article, i would wager the better scores will be found by QA than developers :-P

Re: Even Tetris is hard to test

#39

Post by nanomage »

clebio wrote: post #29
mgraczyk wrote: post #9

Nitpick: As an avid Tetris player, I have to say that the examples given for "extremely rare" events actually happen quite often during normal play. In particular, clearing 4 lines at once, two times in row will generally happen within the first few minutes of game play and is sort of the whole goal (if you're trying to maximize points). Wall kicks are less common but certainly not rare enough that their processing c…

Are there more rare scenarios that you can think of?

* The rotation interactions agains the wall. * The rotation interactions when there are hanging blocks. * Sliding. * Slide rotations.

any others?

Re: Even Tetris is hard to test

#40

Post by BrandonM »

nawitus wrote: post #27
BrandonM wrote: post #22

Earlier quoted context omitted.

I disagree that FP and OOP are the only abstraction methods. Arithmetic can be an excellent abstraction method. For me, it's a code smell to have a function called "updateScore4". It's often possible to come up with an algebraic statement that gives the same result as a bunch of logic (code paths). I think userbinator's point is that better code is easier to test as a result of having fewer code paths. Of course ther…

On the other hand, you can "hide" the different code paths in a single algebraic statement, but even if your tests always execute that statement, you've lost information that you've actually tested the both "paths" of the algebraic statement. I guess it's easier to use a boolean logic example (in JavaScript): return myNormalObject || myDefaultObject(); Your code might always execute the myNormalObject half and return…

That's not arithmetic, though. That's logic represented as a one-liner... Not the sort of change I'm talking about.
Post reply on HN