Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Is TDD or BDD used in AAA gaming industry?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 64 posts

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#51

Post by peruvian »

speps wrote: post #13

If you want to experience that in the AAA game industry, I work at Rare Ltd (UK, part of Microsoft Studios). We've been doing TDD and Continuous Integration for about 5-6 years now starting with Kinect Sports Rivals. For Sea of Thieves[0] (our current project), we've extended that to Continuous Delivery and it's been just a complete game changer in regards to overtime and crunch. For example, the onboarding exercise…

Why did you guys make a new Banjo Kazooie game in 2008 but made it about Lego cars? :-)

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#52

Post by wolfgke »

majewsky wrote: post #49
sk0g wrote: post #42

Earlier quoted context omitted.

That's not TDD, really, but it's a clever way of testing both the AI, and level changes, to a certain extent. If you drop a fucking boulder and you can't cross it yeah, it will light you up with red floodlights, but if a certain corner drops you into the nether... Unless the AI also knows to check each possible interaction, movement, etc.

Can you clarify why you do not consider this TDD?

I think that if one is meticulous, sk0g is actually right (source: https://en.wikipedia.org/w/index.php?title=Test-driven_devel...): In TDD one begins anything (in particular any new feature) with writing a test for it, then one runs all tests to see if something fails and only then starts to write code if necessary until all tests pass.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#53

Post by je42 »

drawkbox wrote: post #22

Game development projects are usually focused on tech/data/network/engine, asset creation/pipeline, behaviors and gameplay. In the tech/data/network/engine areas there may be TDD/BDD if the engine is to run multiple titles beyond the current project, if not the budget may not even allow for time. Possibly other areas may have tests in the behavior/gameplay areas: components, AI etc, this depends on the current dev cu…

> Many times on the production/live side of game development, gameplay and behaviors change so much that any automated testing becomes a lag on dev/iteration time, or aren't updated and stagnate as the changes happen so quickly day to day. This is especially true in iterative development during prototyping or pre/post-production, since it is mostly visual the benefits are outweighed by the speed required to ship.

So you think, you if you do testing you are overall slower ? Skipping tests makes youufaster short term.

Long term, you are slow as hell because of the lack of coverage and manual effort.

Also, especially test help you with large scale changes. Because, these tell you what breaks what shouldn't have been broken.

Further, test is not the same as test.

In order to do efficiently testing you need to apply the test-pyramid.

i.e. have lots of unittest. less integration test. less system tests. less e2e tests.

another guiding principle is to focus automated testing on the things are most curicial for the success of the product, or the most difficult to do manually.

also, you need to chose the tools that support your testing stratgey. if you work against your tooling. oh boy, you have a journey ahead.

(For example unity unittesting framework was pretty crappy last time i check -- 3 years ago )

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#55

Post by falcolas »

hacker_9 wrote: post #40
purrcat259 wrote: post #7

Factorio is a good example of a test driven developed game: https://www.factorio.com/blog/post/fff-186

Is that TDD? Looks like automated tests run after each commit.

I agree; automated integration tests. Still incredibly useful (per the devs), and not quite the industry standard.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#56

Post by Rannath »

zamalek wrote: post #14
dfabulich wrote: post #8

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevit…

This is my impression too. There's also good reasoning: TDD is poorly suited to the industry. For example, how would you test your renderer? Do you run the pipeline and check it against a final image? No, strict TDD would require that the asserted image be hand-crafted. Do you check that each part of the pipeline works as a unit? You can't without running shaders on the GPU (requiring the previous image test). What g…

You don't test your renderer. The renderer is third party tech in 99% of games. You test that the right calls are being made to the renderer. For that you use a mock. Your test are there to assert correctness in YOUR code. if something exists outside your code you test it another way.

Example: Sam is writing an engine. He has OpenGL, DX9, and DX11 bindings. He also has a Mock that logs calls and checks to see that they match known good logs. Sam then sets up a demo to cycle through all the different graphical capabilities of his engine, this also gives Sam the ability to test his AI/scene crafting code.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#57

Post by sk0g »

majewsky wrote: post #49
sk0g wrote: post #42

Earlier quoted context omitted.

That's not TDD, really, but it's a clever way of testing both the AI, and level changes, to a certain extent. If you drop a fucking boulder and you can't cross it yeah, it will light you up with red floodlights, but if a certain corner drops you into the nether... Unless the AI also knows to check each possible interaction, movement, etc.

Can you clarify why you do not consider this TDD?

In TDD you only write code if it makes a red light green, so to speak.

First you make the test, then you run it. It will obviously fail, and then you implement code that starts passing tests.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#58

Post by EnderMB »

shubhamjain wrote: post #31
dfabulich wrote: post #8

In my experience, no, they don't write tests first, and in fact AAA game companies write almost no automated tests at all. Instead, AAA teams have large staffs of cheap manual testers--high school and college kids who think they're getting paid to play video games. The managers in charge of these QA teams work their manual testers to the bone, encouraging them to work late and do unpaid overtime. These testers inevit…

I find it hard to wrap my head around how the whole AAA VG ecosystem works. How is it able to function? AAA Video Games are highly complex engineering projects but, as I understand, under the hood you have naive and inexperienced kids, horrible engineering practices, and a chaotic work environment. How does this industry manage to hire so many quality engineers and treat them as dispensable that easily?

A friend of mine has worked on AAA games in the past, and although he says it's fairly hard work, there is always work out there.

Also, he likes to chop and change every now and then, and the norm seems to be that once a company delivers a game the vast majority of the development team is made redundant. He takes the redundancy, moves to another company/game, and repeats this cycle.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#59

Post by drawkbox »

je42 wrote: post #53
drawkbox wrote: post #22

Game development projects are usually focused on tech/data/network/engine, asset creation/pipeline, behaviors and gameplay. In the tech/data/network/engine areas there may be TDD/BDD if the engine is to run multiple titles beyond the current project, if not the budget may not even allow for time. Possibly other areas may have tests in the behavior/gameplay areas: components, AI etc, this depends on the current dev cu…

> Many times on the production/live side of game development, gameplay and behaviors change so much that any automated testing becomes a lag on dev/iteration time, or aren't updated and stagnate as the changes happen so quickly day to day. This is especially true in iterative development during prototyping or pre/post-production, since it is mostly visual the benefits are outweighed by the speed required to ship. So…

> So you think, you if you do testing you are overall slower ? Skipping tests makes you faster short term.

Not at all long run, but in the prototyping stage it may be a hindrance or when you are going for a fun gameplay mechanic over coverage. When you lock something down then tests make sense in this scenario. You'd always want tests for unit testing data, network, core tech, but might not need it when you are building gameplay that the test is really a user experiencing it/tuning it. Even writing some rendering or gameplay tests it takes time to get them right before they are actually useful tests.

Agree on all the other items there are many levels of testing. Usually core libraries are covered, when it comes to in-progress prototyping/pre-production, it can definitely slow you down (by adding weight to changes) if it is changing constantly and not yet locked down. Another area that weighs into it is how much budget and time that the code might be in use, if it is just a tool or something that isn't needed for live code (something prototyping or to help develop it, i.e. asset packing or the like) it might not be worth it to surround it with unit/integration testing.

Re: Ask HN: Is TDD or BDD used in AAA gaming industry?

#60

Post by existencebox »

Negitivefrags wrote: post #39

I work on a game called Path of Exile. At our studio we don't do TDD, but we do continuous integration with a lot of asset testing. In the early days, we tried doing unit tests, but honestly, it's very hard in a game. We had little success testing gameplay code. What we have a huge amount of is asset tests. We have 4 times more non-programmers than programmers working on the game, so most of what is created isn't cod…

Let me take this opportunity to say "you guys do some damn good work". Certainly one of the more impressive games I play, stability/longevity/feature growth wise, especially considering the size of GGG.

Topical question; however, since the thread on a whole has focused on the more visible but more finniky-to-test aspects of these games. Do your same statements apply more or less to the backend/networking infra? I remember the decent ball of work you guys wrote about a while back when introducing the lockstep change, was curious what went on behind the scenes to support and validate that from an ops/maintainability perspective, if you can speak on that at all.

Post reply on HN