Ask HN: Is TDD or BDD used in AAA gaming industry? • Hacker News Board

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#61

Post by grogenaut » Wed, Jul 19, 2017, 6:13 PM UTC

I worked on a AAA game. I controlled the networking code (web services really) down and the entire server infra for our project. All of that was full TDD but it was also all greenfield by me who had done heavy TDD before and working with an external team who was all TDD. I even wrote integration tests from a stub client that ran on the playstation as part of the build to hit my server and a stub server to do wire testing for the client as part of my builds. Also had a full load regression test as part of my build that ran every checkin on a smaller slower cluster.

The main game had no tests... except that you need to realize that we had 480GB of asset compiler integration tests and 32GB of game engine integration tests.

As a heavy TDD person it took me a long time to realize that this can be very sufficient though it's a longer turn around loop. The test coverage just from being able to boot the game up is quite large. This is something that many non-game type programs have real issues with, they can't easily just "run all the user data". I know we had one project where we did a lot of TDD but it wasn't actually as good as just running the entire input set every build which I eventually switched us to. This was a non-game data processing system. Doing this + unit testing gave us great coverage and few bugs.

Generally major bugs were found within 15 minutes by another engineer, eg the next person to sync, build or run the engine or art compiler.

I do think we could have unit tested the core collections libraries in the engine to great effect and faster turn around but the build was not really segregated in a way that we would take advantage of this. It was just a monolithic build; the collections objects were heavily header included so there really would have been a build step to built them into their own test exe, run them then build the game. So I'm not sure how much benefit we would have gotten. I think in 4 years I saw one collections issue that lasted past just launching the engine and not having it crash.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#62

Post by ajeet_dhaliwal » Wed, Jul 19, 2017, 6:15 PM UTC

discreteevent wrote: post #29

zamalek wrote: post #14

Earlier quoted context omitted.

This is my impression too. There's also good reasoning: TDD is poorly suited to the industry. For example, how would you test your renderer? Do you run the pipeline and check it against a final image? No, strict TDD would require that the asserted image be hand-crafted. Do you check that each part of the pipeline works as a unit? You can't without running shaders on the GPU (requiring the previous image test). What g…

In my opinion it's the integration tests that provide 99.99% of the value in 99.99% of the cases. Maybe not full integration tests (which can be expensive to run) but at least subsystem tests. These are the ones that find the real bugs. A very high percentage of unit tests once passing tend to continue to pass and only fail when you intentionally change the implementation. i.e. they impede your progress. Low level un…

Exactly my experience too. I have been an automation developer in three teams across two games companies - integration tests provide great value if run continuously and daily, they are especially helpful for uncovering regression. Also the parent describing the sad state of affairs in the AAA games industry is fairly accurate and that leaves little hope for TDD/BDD and other similar practices. One of the sources of all of these problems are business model challenges at a company level which then leads to low pay and high stress creating high attrition. Things just aren't planned out the way you may think they would be. The toss away tradition means that unlike companies developing long running services where automated tests written once may be valuable for even a DECADE, that's not true in games so the ROI isn't quite as high, although this may be changing slightly at least when it comes to engine churn and so in does appear to be slowing down.

Aside - having worked on automated testing in the AAA games industry I have found that good reporting helped managers and other devs better see the ROI from the work put in and I have been involved in creating a service for reporting automated test results across dev teams, see my profile if you're interested and give it a try, you can use it for free and I'd love to get your feedback - send me an email.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#63

Post by Negitivefrags » Wed, Jul 19, 2017, 9:18 PM UTC

existencebox wrote: post #60

Negitivefrags wrote: post #39

I work on a game called Path of Exile. At our studio we don't do TDD, but we do continuous integration with a lot of asset testing. In the early days, we tried doing unit tests, but honestly, it's very hard in a game. We had little success testing gameplay code. What we have a huge amount of is asset tests. We have 4 times more non-programmers than programmers working on the game, so most of what is created isn't cod…

Let me take this opportunity to say "you guys do some damn good work". Certainly one of the more impressive games I play, stability/longevity/feature growth wise, especially considering the size of GGG. Topical question; however, since the thread on a whole has focused on the more visible but more finniky-to-test aspects of these games. Do your same statements apply more or less to the backend/networking infra? I rem…

For backend we test with a load testing bot, but we don't do it every commit, just after making larger changes to the backend. I would like to test more, and I'm actually working on better automation for load testing at the moment so that it's easier for us to run them.

Testing lockstep was quite an endeavour. The approach I used was to add a very verbose log to the client and the server which logs basically every gameplay relevant decision. That includes moving a single unit, updating a stat, etc.

Then every time a line is logged, we update a hash, and then send the hash to the client. The client, being in lockstep, should be able to come up with the same hash. If the hash is different, then the client and server both break and we can then analyse the state and see what is different.

As we discovered issues, we added more and more log lines and fixed bugs until everything ran perfectly the same on the client and server.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#64

Post by samplonius » Wed, Jul 19, 2017, 9:30 PM UTC

user5994461 wrote: post #32

shubhamjain wrote: post #31

Earlier quoted context omitted.

I find it hard to wrap my head around how the whole AAA VG ecosystem works. How is it able to function? AAA Video Games are highly complex engineering projects but, as I understand, under the hood you have naive and inexperienced kids, horrible engineering practices, and a chaotic work environment. How does this industry manage to hire so many quality engineers and treat them as dispensable that easily?

Because games are cool and there is no shortage of good developers. All games are throwaway projects. Write one, sell for a year, gone. There is no maintenance period.

Except for DLC. A AAA title will need to ship at least 4 x DLCs over a 12 to 18 month period. And no doubt the DLC development will expose serious bugs along the way, so the DLC will have to contain patches as well.

The lifespan of AAA titles is longer now that most are purchased digitally.