Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Even Tetris is hard to test

blog.jwhitham.org

Post reply on HN
Print view View original

11–20 of 42 posts

Re: Even Tetris is hard to test

#11

Post by aikah »

canadev wrote: post #6
lucaspiller wrote: post #4

Earlier quoted context omitted.

One of the things I've found as a side effect of people using code coverage tools is that instead of testing the behaviour of a method they end up testing the implementation. I think this is because they initially test the behaviour, but then see that one path is missing, so add a test to ensure that path is ran - instead of just testing the behaviour that calls that path and checking the code coverage tool. This end…

I've faced this problem in my own tests. I want to achieve total coverage so that I know that I've got all the cases covered, but then I end up testing the implementation rather than the contract. I'm not sure what to do about it.

You cant have it both ways in my opinion. High test coverage == testing every possible paths == looking at implementation details. If you are testing an algorithm(and games are full of these)you want it to be a 100% accurate therefore you dont have much choice.

Re: Even Tetris is hard to test

#12

Post by rrrrob »

forrestthewoods wrote: post #2

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

For example, 100% coverage won't help with the classic Windows Tetris bug where the score overflows at 32768.

Re: Even Tetris is hard to test

#13

Post by thu »

The interesting bit that isn't mentioned is that all the given examples are actually part of the game specification, i.e. those behaviours are there on purpose. It means that if you have an accurate list of the desired features, you could probably also achieve 100% coverage. It is also possible that testing some features would not increase the code coverage.

"Hard to test" in the submission title didn't mean what I thought: I though it meant it was hard to write tests for Tetris, not that it was hard to recover a complete specification of the game while playing it.

Re: Even Tetris is hard to test

#14

Post by canadev »

aikah wrote: post #11
canadev wrote: post #6

Earlier quoted context omitted.

I've faced this problem in my own tests. I want to achieve total coverage so that I know that I've got all the cases covered, but then I end up testing the implementation rather than the contract. I'm not sure what to do about it.

You cant have it both ways in my opinion. High test coverage == testing every possible paths == looking at implementation details. If you are testing an algorithm(and games are full of these)you want it to be a 100% accurate therefore you dont have much choice.

This is sort of the conclusion I was coming to. I am glad to hear someone else express it :)

Re: Even Tetris is hard to test

#15

Post by Joeri »

thu wrote: post #13

The interesting bit that isn't mentioned is that all the given examples are actually part of the game specification, i.e. those behaviours are there on purpose. It means that if you have an accurate list of the desired features, you could probably also achieve 100% coverage. It is also possible that testing some features would not increase the code coverage. "Hard to test" in the submission title didn't mean what I t…

The corrolary is that you must have a specification in order to test comprehensively. Either you use some form of TDD and your tests are your specification (and any behavior not under test is undefined), or you have exact and complete specifications that the as-built software can be compared to, and any hehavior not in the specficiation is undefined.

Re: Even Tetris is hard to test

#16

Post by xxs »

rrrrob wrote: post #12
forrestthewoods wrote: post #2

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

For example, 100% coverage won't help with the classic Windows Tetris bug where the score overflows at 32768.

Yes, overflows and data races are bane for the coverage tests - just a false sense of security.

Re: Even Tetris is hard to test

#17

Post by ygra »

forrestthewoods wrote: post #2

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

  »Our customers tend to be makers of aircraft or car
   parts. Both businesses have strict safety standards
   which involve coverage testing, and our tools help you
   produce the relevant reports for certification, like
   DO-178B for the aviation industry.«
I guess in both industries the value of more testing cannot be understated.

Re: Even Tetris is hard to test

#18

Post by singingfish »

Bah one line of BBC basic. One line of incomprehensible BBC basic more like. When I was at school I implemented a game in three lines of spectrum basic. 5 if you wanted scoring. And it was readable. And it was the most popular computer game in the school by virtue of it being quicker to type than loading a tape, and being moderately entertaining. Just bound the cursor keys to the values in the line() function. The game crashed if the line went out of bounds :).

Re: Even Tetris is hard to test

#19

Post by alkonaut »

forrestthewoods wrote: post #2

Interesting. I do wonder what the actual value of 100% coverage. I'm not saying it's not substantial, I'm just curious how many cases that still misses. There are a lot of permutations of data that can be used by code, how many are possible and how many cause issues?

I have yet to see a compelling argument that 100% test coverage would be "better" than 50% without knowing what the quality of the tests are In those cases. I'm always suspicious of code bases that claim 100% test coverage because they usually have it because someone has decided it must have, which invariably means it has tests that exist only to execute some corner case lines of code.

To have good test coverage you should both test all possible inputs and have proper asserts for those inputs.

Testing is hard, covering lines of code isn't. To put it another way: in a test the hard bit is the assert, not the call into the tested code. And coverage only reports on the former.

Re: Even Tetris is hard to test

#20

Post by alkonaut »

aikah wrote: post #11
canadev wrote: post #6

Earlier quoted context omitted.

I've faced this problem in my own tests. I want to achieve total coverage so that I know that I've got all the cases covered, but then I end up testing the implementation rather than the contract. I'm not sure what to do about it.

You cant have it both ways in my opinion. High test coverage == testing every possible paths == looking at implementation details. If you are testing an algorithm(and games are full of these)you want it to be a 100% accurate therefore you dont have much choice.

Tests should be based on the specification. If I want to change some internal implementation detail I should only have to verify that the current tests pass.

If a e.g game contains a sorting in some place in the renderer, I can replace the quicksort with a mergesort as long as the renderer interface is still testing ok. The new sort algorithm may have new special case paths (even number of items vs odd for example) but it's not a concern of the renderer public interface. I may however have introduced a bug with an odd number of items here and the old code was 100% covered and now it isn't. So there is a potential problem and the 99% has actually helped spot it.

If the sorting is a private implementation detail of the renderer then there is no other place to test it than to add a new test to the renderer component only because the sorting algo requires it for a code path. This is BAD.

The proper action here is NOT to add tests to the renderer component to test the sorting code path, but instead to make the sorting visible and testable in isolation via its own public interface.

So one of the positive things about requiring coverage is that if you do it right, it will lead to smaller and more decoupled modules of code.

The bad thing is that if you do it wrong you will have your God classes and a bunch of tests coupled tightly to them.

Post reply on HN