Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is TDD or BDD used in AAA gaming industry?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 64 posts

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#21

Post by cbanek »

teknologist wrote: post #17
dfabulich wrote: post #8

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevit…

> When manual QA is that cheap, it turns the economics of automated testing on its head. Why maintain an automated regression suite when you have humans who will do the testing for you? But in practice those who aren't paid a lot don't particularly care much about thorough testing. It requires a systematic approach that a kid out of college who just wants to get paid playing video games isn't going to develop

They aren't just sitting there playing the games, or designing test cases (usually) - there are actual documents laying out what the test case is and what the manual tester should do to exercise the case. These cases are split up usually among a room of people who check them off one by one for a given build under test.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#22

Post by drawkbox »

Game development projects are usually focused on tech/data/network/engine, asset creation/pipeline, behaviors and gameplay.

In the tech/data/network/engine areas there may be TDD/BDD if the engine is to run multiple titles beyond the current project, if not the budget may not even allow for time. Possibly other areas may have tests in the behavior/gameplay areas: components, AI etc, this depends on the current dev culture usually.

Really the main thing in game development is shipping and building fun, if tests help you do that faster/better then you do them. I think most devs developing anything that is not game specific but tech/data/network/engine related likes to have tests to rely on, time permitting.

Most studios I have worked with that do some TDD/BDD/automated testing it usually happens on the "tech" or core/engine teams rather than the "production" or "live" teams. The tech teams are working on engine/network/data while the live/production teams are using the engine/tools/scripts to make assets and gameplay. The tech team isn't as locked to game launch dates/crunches and has more time to be thorough because most of their code will last across multiple titles.

For the most part, in the asset creation/pipeline, behaviors and gameplay area, more of that is visual and QA based. There are testing companies and departments setup just for the gameplay/feature testing. So much of game development is going for fun and a good mechanic that actually playing it and testing it live is more effective than automated tests in some areas. Although there are some automated test tools in Unity[1], Unreal[2] and custom engines maybe for very rote areas of gameplay (physics engine, collectables, asset loading, very common unchanging actions etc).

Many times on the production/live side of game development, gameplay and behaviors change so much that any automated testing becomes a lag on dev/iteration time, or aren't updated and stagnate as the changes happen so quickly day to day. This is especially true in iterative development during prototyping or pre/post-production, since it is mostly visual the benefits are outweighed by the speed required to ship.

[1] https://unity3d.com/unity/qa/test-tools

[2] https://docs.unrealengine.com/latest/INT/Programming/Automat...

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#23

Post by speps »

SCdF wrote: post #19
speps wrote: post #13

If you want to experience that in the AAA game industry, I work at Rare Ltd (UK, part of Microsoft Studios). We've been doing TDD and Continuous Integration for about 5-6 years now starting with Kinect Sports Rivals. For Sea of Thieves[0] (our current project), we've extended that to Continuous Delivery and it's been just a complete game changer in regards to overtime and crunch. For example, the onboarding exercise…

Can you give some examples of what you're actually testing? Perhaps an example of a specific test statement, the kind of level the tests sit at etc, what gets covered and what doesn't etc? Are you testing the rendering engine? Lighting / physics / shaders / collision detect? Are you testing scripting? Win conditions? AI (probably not for SoT I guess)? Are you testing the core engine? File mgmt? memory mgmt? input rea…

We are unit testing all new code that can be unit tested, trying to do dependency injection when possible so mocking is easier.

We are testing the rendering engine, lighting, shaders, etc. in a visual test framework we created ourselves that compares each run of the tests with the previous and uses SSIM to output a comparison metric. At the moment, that's still checked by someone every week but could be automated. However, when changing rendering code or shaders, we do require that a test result is shown during code review.

For physics and collision detection, we do have tests covering that too, a lot of them involving our FFT water simulation as well.

We are testing the whole flow of the game from start menu to entering a session, matchmaking, loading a map etc. I think AI also has some good coverage, notably for the navmesh which is essential.

The core engine is Unreal Engine 4. It's not an engine made for testing unfortunately, even though the vanilla version has some (but not nearly enough). We've stuck with a specific version for a while now as upgrading usually meant that we'd also get new bugs from the new version. All the core is covered by all the existing tests, we've added tests where we thought it was critical and where we fixed or improved internal code of the engine.

Before joining Rare, I really wasn't aware of any testing whatsoever in the video games industry. It's been an eye opener and I wouldn't do it any other way.

Reading other comments, we might not follow TDD all the time or to the letter but the point is that testing itself is a very part of everything we do as engineers. That allows our QA team to focus on stuff that is actually broken and can't be tested (such as if a feature is actually fun).

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#24

Post by Nysa »

I am not working directly in gamedev but I have some insights.

I've seen growing interest in automated testing from mobile apps.

Basically guys were writing regression testing with framework like Selenium (UI testing) that goes throughout most common scenarios. (I think that was described by huuugegames.com on some presentation)

The main idea was to let QA focus on "exploratory" tests. (Testing rules of the game, design decisions or performance)

TDD & BDD can be used gamedev but not in places you will think of first.

I will simplify it vastly: programmers make tools for designers to use.

Tools are made by programmers - tools can be done with TDD and/or BDD. Games are made by designer - they use tools to make rules, systems etc.

Therefore gameplay is make mostly out of data (designer's parameters set in engine/tools) - you can't test your data with TDD/BDD.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#25

Post by Nysa »

I am not working directly in gamedev but I have some insights.

I've seen growing interest in automated testing from mobile apps.

Basically guys were writing regression testing with framework like Selenium (UI testing) that goes throughout most common scenarios. (I think that was described by huuugegames.com on some presentation)

The main idea was to let QA focus on "exploratory" tests. (Testing rules of the game, design decisions or performance)

TDD & BDD can be used gamedev but not in places you will think of first.

I will simplify it vastly: programmers make tools for designers to use.

Tools are made by programmers - tools can be done with TDD and/or BDD. Games are made by designer - they use tools to make rules, systems etc.

Therefore gameplay is make mostly out of data (designer's parameters set in engine/tools) - you can't test your data with TDD/BDD.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#26

Post by SCdF »

speps wrote: post #23
SCdF wrote: post #19

Earlier quoted context omitted.

Can you give some examples of what you're actually testing? Perhaps an example of a specific test statement, the kind of level the tests sit at etc, what gets covered and what doesn't etc? Are you testing the rendering engine? Lighting / physics / shaders / collision detect? Are you testing scripting? Win conditions? AI (probably not for SoT I guess)? Are you testing the core engine? File mgmt? memory mgmt? input rea…

We are unit testing all new code that can be unit tested, trying to do dependency injection when possible so mocking is easier. We are testing the rendering engine, lighting, shaders, etc. in a visual test framework we created ourselves that compares each run of the tests with the previous and uses SSIM to output a comparison metric. At the moment, that's still checked by someone every week but could be automated. Ho…

Thanks for your reply.

> We are testing the rendering engine, lighting, shaders, etc. in a visual test framework we created ourselves that compares each run of the tests with the previous and uses SSIM to output a comparison metric

So would this be something like rendering a frame of a test scene (utah teapot? real in-game scene?) in various states of rendering (with / without lighting, with / without specular / normals etc), and then making sure they look the same each time?

> Before joining Rare, I really wasn't aware of any testing whatsoever in the video games industry. It's been an eye opener and I wouldn't do it any other way.

You should definitely consider a GDC/etc talk, I imagine it would be an eye opener for a lot of people.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#27

Post by wolfgke »

Croteam applied lots of automated testing by bots in The Talos Principle to ensure that changes in level design cannot lead to a situation where the game cannot be completed anymore:

> https://venturebeat.com/2014/11/04/the-talos-principle-under...

> http://www.gdcvault.com/play/1022784/Fast-Iteration-Tools-in...

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#28

Post by speps »

SCdF wrote: post #26
speps wrote: post #23

Earlier quoted context omitted.

We are unit testing all new code that can be unit tested, trying to do dependency injection when possible so mocking is easier. We are testing the rendering engine, lighting, shaders, etc. in a visual test framework we created ourselves that compares each run of the tests with the previous and uses SSIM to output a comparison metric. At the moment, that's still checked by someone every week but could be automated. Ho…

Thanks for your reply. > We are testing the rendering engine, lighting, shaders, etc. in a visual test framework we created ourselves that compares each run of the tests with the previous and uses SSIM to output a comparison metric So would this be something like rendering a frame of a test scene (utah teapot? real in-game scene?) in various states of rendering (with / without lighting, with / without specular / norm…

> So would this be something like rendering a frame of a test scene (utah teapot? real in-game scene?) in various states of rendering (with / without lighting, with / without specular / normals etc), and then making sure they look the same each time?

Yes, exactly, it's small test scenes we setup ourselves whenever we add a new rendering feature or fix a rendering bug. Some of them are still not deterministic as we're using Compute Shaders for some stuff which is hard to make deterministic.

> You should definitely consider a GDC/etc talk, I imagine it would be an eye opener for a lot of people.

We are currently looking for topics. We'll see what comes out of that discussion internally :)

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#29

Post by discreteevent »

zamalek wrote: post #14
dfabulich wrote: post #8

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevit…

This is my impression too. There's also good reasoning: TDD is poorly suited to the industry. For example, how would you test your renderer? Do you run the pipeline and check it against a final image? No, strict TDD would require that the asserted image be hand-crafted. Do you check that each part of the pipeline works as a unit? You can't without running shaders on the GPU (requiring the previous image test). What g…

In my opinion it's the integration tests that provide 99.99% of the value in 99.99% of the cases. Maybe not full integration tests (which can be expensive to run) but at least subsystem tests. These are the ones that find the real bugs. A very high percentage of unit tests once passing tend to continue to pass and only fail when you intentionally change the implementation. i.e. they impede your progress. Low level unit tests are very good for testing an algorithm or something like a parser, say, but that's not a high percentage of the code on the systems that I have worked on.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#30

Post by maccard »

ZenoArrow wrote: post #6

I'm not in the game industry, but from what little I know, this is probably going to be a tricky question to answer. Reason being, a high proportion of AAA game development is done using off-the-shelf game engines. Of course it's still necessary to write code for the game logic and features, but because of the stable base this code is written against, it might not be seen as necessary to follow a strict TDD/BDD appro…

> a high proportion of AAA game development is done using off-the-shelf game engines.

I don't think this is particularly true. A high proportion of game development is, but many AAA studios are using either home-rolled engines, or started with an existing engine (Unreal or iDTech) and what they're currently running looks absolutely nothing like the original engine.

Post reply on HN