Live data from Hacker News

Ask HN: Is TDD or BDD used in AAA gaming industry?

news.ycombinator.com

11–20 of 64 posts

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#11
Short answer: No. Gamedevs have to keep showing testing zealots the door.

Long answer: The problems with the approach are manyfold. There are a few points where testing is automatable, but they don't describe entire game projects well.

* What are you testing when you add gameplay? You are testing for a whole set of design concerns across the project, not just a technical specification or quality of service metric. You cannot do it in isolation and get the results you want at the speed you need, because you need tons of feedback to discover what a complex game is currently strong or weak at. The only reasonable way to gain the necessary feedback is to allow prototyping code to drive near-term changes, manually playtest it, and then factor out the most stable parts of the resulting soup where you can. This works against test-first because you don't really know what you're testing. You throw in a feature on a hunch - "maybe this will give the experience we're looking for" - and see how it behaves hands-on. If it works, there are still usually ramifications and elaborations that didn't surface up-front. And once it does work it's hard to break out into a sandbox because the interesting part of the behavior is in the coupling to other systems and features.

* Games deal with vast quantities of mutable state strapped onto relatively straightforward data pipelines: the parts that are most testable are engine and toolchain core elements like asset builds and rendering systems. These parts do see unit tests and integration tests, although there isn't total consensus on how much testing is needed or how it should be done. They are the most like business and application software, though, since their development can be more structured around technical goals and quality metrics.

* Many bugs are ultimately data bugs. Some designer or artist set up the wrong binding of assets or made a valid yet broken combination of parameters. Games tend to fly apart very suddenly when they operate on bad data. But the code is doing exactly what it should do for that set of inputs - and if it's a failure it's at the level of process and tooling, not the runtime code.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#12

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevit…

In addition to the steady supply of college-age fools who think that life as a video game tester would be fulfilling; it is also borderline impossible to test many things in video games adequately without human intervention.

Aside from critical technical elements like netplay protocol and file parsing, the only reliable signal you'll get from automated testing is whether the code crashes or not under normal circumstances on the test platform with the tested driver and operating system patch release, most properties of the game are subjective or emergent from the code itself.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#13
If you want to experience that in the AAA game industry, I work at Rare Ltd (UK, part of Microsoft Studios). We've been doing TDD and Continuous Integration for about 5-6 years now starting with Kinect Sports Rivals. For Sea of Thieves[0] (our current project), we've extended that to Continuous Delivery and it's been just a complete game changer in regards to overtime and crunch. For example, the onboarding exercise is basically a TDD kata that has to use our coding guidelines and build farm so you get to know how we work without having to dive right away into Unreal Engine 4. We use TeamCity for our CI solution.

And yes, we are recruiting : https://www.rare.co.uk/careers

[0]: https://www.seaofthieves.com

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#14

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevit…

This is my impression too. There's also good reasoning: TDD is poorly suited to the industry.

For example, how would you test your renderer? Do you run the pipeline and check it against a final image? No, strict TDD would require that the asserted image be hand-crafted. Do you check that each part of the pipeline works as a unit? You can't without running shaders on the GPU (requiring the previous image test).

What game developers are slowly starting to master (already far more than anyone else) is integration testing. Instead of requiring detailed repro steps, just record the testing session in extreme detail and replay it - if anything goes wrong this can be turned into an automated test case.

There are no golden hammers and that includes TDD. TDD is an amazing solution that can provide value in 99.99% of scenarios. We should be scrutinizing areas where it doesn't add value; there is interesting stuff going on there.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#15
My biggest observation with AAA games is that the coding is actually a fairly small part of the work. Most of it is one huge asset pipeline. And I don't think we have particularly good tech for testing what "looks right". Maybe computer vision will get that good eventually, but it's not even a thing right now. The same applies to the gross properties of renderers.

Even in big-spec-up-front triple-A games, there's a lot of exploratory work where you honestly don't know what you want. BDD and TDD don't help there either, rapid prototyping does. (TDD presupposes you know what you want, BDD presupposes someone does and you need to find and talk to them.)

None of which is to say that there aren't many places where test friendly design and regression tests wouldn't help. But the surface isn't the same as the average business app.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#16

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevit…

In addition to the steady supply of college-age fools who think that life as a video game tester would be fulfilling; it is also borderline impossible to test many things in video games adequately without human intervention. Aside from critical technical elements like netplay protocol and file parsing, the only reliable signal you'll get from automated testing is whether the code crashes or not under normal circumsta…

This is true at a blackbox level but you can pretty effectively create automated tests of finer grained software components. The idea is presumably that if you have a bunch of components that work well, the overall system will tend to work better.

I think this is generally true although it has limits; the higher up you go in the stack, the more value the test has. Still, if it's prohibitively difficult to test higher up the stack, might as well test what you can, no?

I found from my years in at EA in the early 2000s that no matter what the technical issues, the absence of automated testing was at the very least a significant cultural issue. Engineers just didn't think about unit tests. Sounds like that hasn't changed.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#17

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevit…

> When manual QA is that cheap, it turns the economics of automated testing on its head. Why maintain an automated regression suite when you have humans who will do the testing for you?

But in practice those who aren't paid a lot don't particularly care much about thorough testing. It requires a systematic approach that a kid out of college who just wants to get paid playing video games isn't going to develop

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#18

I'm not in the game industry, but from what little I know, this is probably going to be a tricky question to answer. Reason being, a high proportion of AAA game development is done using off-the-shelf game engines. Of course it's still necessary to write code for the game logic and features, but because of the stable base this code is written against, it might not be seen as necessary to follow a strict TDD/BDD appro…

A high proportion of web development is done using somehow "off-the-shelf" web frameworks, and yet people write unit test/functional tests for the application logic and features. Despite how stable the base this code is written against.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#19
post #13

If you want to experience that in the AAA game industry, I work at Rare Ltd (UK, part of Microsoft Studios). We've been doing TDD and Continuous Integration for about 5-6 years now starting with Kinect Sports Rivals. For Sea of Thieves[0] (our current project), we've extended that to Continuous Delivery and it's been just a complete game changer in regards to overtime and crunch. For example, the onboarding exercise…

Can you give some examples of what you're actually testing? Perhaps an example of a specific test statement, the kind of level the tests sit at etc, what gets covered and what doesn't etc?

Are you testing the rendering engine? Lighting / physics / shaders / collision detect?

Are you testing scripting? Win conditions? AI (probably not for SoT I guess)?

Are you testing the core engine? File mgmt? memory mgmt? input reading etc?

As someone who is a developer and a gamer but has never combined the two, the idea of TDD in that scenario is fascinating.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#20

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevit…

This echos my experience completely.

One other thing to realize is that you're building a game, you're not building tech. The tech supports the game. A buggy game can be fixed, but a game that works but isn't fun is hard to mend.

A lot of the game is also in the UI, which is notoriously hard to test, let alone use TDD.

Post reply on HN