> why game devs don't unit test
Sources ?
Addition with flamethrowers – why game devs don't unit test
31–40 of 96 posts
Re: Addition with flamethrowers – why game devs don't unit test
#32I don't buy this argument. Most game developers I know have said that unit tests are a waste of time so they never use them, but they're struggling with making changes to utility code and making sure that it doesn't do the wrong thing. Y'know, what unit tests are for. I think the key here is that the perceived cost / benefit ratio is too high. It's the perception that drives their behavior though. I'm in a company no…
Valve became serious about software quality in Dota 2 around 2017 - about 7 years after launch. Before that game updates were accompanied with lots of bugs that would take weeks to fix. These days, there are still tons of bugs, but much better than before. They just released one of the biggest updates in the game's history this week, and there are hardly any bugs being reported. I am pretty sure there is some sort of…
Reminds me of an article about the testing infrastructure of League and Legends [1] back in 2016. 5500 tests per build in 1 to 2 hours.
Games are extremely hard to test. For me it falls into the same category like GUI testing frameworks which imho are extremely annoying and brittle. Except that games are comparable to a user interface consisting of many buttons which you can short and long press and drag around while at the same time other bots are pressing the same buttons, sharing the same state influenced by a physics engine.
How do you test such a ball of mud which also constantly changes by devs trying to follow the fun? Yes you can unittest individual, reusable parts. But integration tests, which require large, time sensitive modules, all strapped together and running at the same time? It's mindboggling hard.
Moreover if you're in a conceptual phase of development and prototyping and idea, tests make no sense. The requirements change all the time and complex tests hold you back. But the funny thing is, that game development stays in that phase most of the time. And when the game is done, you start a new one with a completely different set of requirements.
There are exceptions, like League of Legends. The game left the conceptual phase many years ago and its rules are set in stone. And a game which runs successfully for that long is super rare.
[1] https://technology.riotgames.com/news/automated-testing-leag...
Re: Addition with flamethrowers – why game devs don't unit test
#33Arguments against testing tend to fall prey to the von Neumann Objection: they insist there is something tests can’t catch, and then they tell you precisely what it is that tests can’t catch… so you can always imagine writing tests for that specific thing. E.g. this article uses an example of removing the number 5, causing the developer to have to implement a base-9 numbering system. Unit tests that confirm this cust…
Tests can't catch race conditions in multithreaded code. Now that I told you what the tests can't catch, can you imagine writing tests for that specific thing?
I've written tests around multithreaded code, but they typically catch them in a statistical manner - either running a bit of code many times over to try and catch an edge condition, or by overloading the system to persuade rarer orderings to occur.
There's also https://clang.llvm.org/docs/ThreadSafetyAnalysis.html which can statically catch some threading issues, though I've not used it much myself.
Re: Addition with flamethrowers – why game devs don't unit test
#34I don't buy this argument. Most game developers I know have said that unit tests are a waste of time so they never use them, but they're struggling with making changes to utility code and making sure that it doesn't do the wrong thing. Y'know, what unit tests are for. I think the key here is that the perceived cost / benefit ratio is too high. It's the perception that drives their behavior though. I'm in a company no…
I’ve seen people slog through untested code where they fear to make a change but I’ve also seen people slog through code with too much test coverage where the tests go through constant churn.
I don’t understand why people don’t just add one test even if the codebase otherwise has zero tests if they’re so scared of one area and I don’t get why people keep adding excessive coverage if it’s wasting their time.
It’s like people pick a stance and then stick with it forever when I couldn’t care less how I’ve been doing something for 10 years if today you showed me a better way.
Re: Addition with flamethrowers – why game devs don't unit test
#35Earlier quoted context omitted.
We produce a library that gets included in software made by our clients, and we have several thousand clients. The uptake on new releases is low (most of the clients believe in "if it ain't broke, don't fix it"). So every release has the potential to live in the wild and need support for a long time. We're also in an industry with a ton of competitors. On top of that, the company was founded by some very junior engin…
How do you stay sane working with clowns?
I get paid good money to work with clowns
Re: Addition with flamethrowers – why game devs don't unit test
#36I don't buy this argument. Most game developers I know have said that unit tests are a waste of time so they never use them, but they're struggling with making changes to utility code and making sure that it doesn't do the wrong thing. Y'know, what unit tests are for. I think the key here is that the perceived cost / benefit ratio is too high. It's the perception that drives their behavior though. I'm in a company no…
> Also, remember that games are not very long-lived pieces of software. You build it, release it, maybe patch it, and move on. This was true a couple decades ago. Nowadays many games are cash cows for decades. Path of Exile was released in 2013, Minecraft in 2011, and World of Warcraft in 2004, and all of those continue to receive regular updates (and have over the course of their lives) and still make plenty of mone…
You can add rigor to your decade-plus cash cow later, once it’s clear that you’ve hit the jackpot.
Re: Addition with flamethrowers – why game devs don't unit test
#37Earlier quoted context omitted.
Valve became serious about software quality in Dota 2 around 2017 - about 7 years after launch. Before that game updates were accompanied with lots of bugs that would take weeks to fix. These days, there are still tons of bugs, but much better than before. They just released one of the biggest updates in the game's history this week, and there are hardly any bugs being reported. I am pretty sure there is some sort of…
Reminds me of an article about the testing infrastructure of League and Legends [1] back in 2016. 5500 tests per build in 1 to 2 hours. Games are extremely hard to test. For me it falls into the same category like GUI testing frameworks which imho are extremely annoying and brittle. Except that games are comparable to a user interface consisting of many buttons which you can short and long press and drag around while…
I doubt Dota 2 devs are writing code like this to test. The game is far too complicated, even more so than league, and changes a lot over the years, for this to be viable.
Dota 2 and openai had a collaboration in 2018ish, and during this time the Dota 2 bots system was reworked completely. They already can generate videos of every spell in action [1], and I would assume this is done by asking AI bots to demonstrate the spell. My guess is that before pushing out an update, a human looks at these videos and other more complex interaction videos for every major change, along with relevant numbers (damage, healing, movement speed), and see if everything makes sense.
I think this, because a lot of times recently, changes in one hero often cause an un-updated hero to break, because they had some backend similarity. And the patch is released with the bug.
Then again, there is no public info, so all the above are wild speculations.
[1] example https://www.dota2.com/hero/treantprotector
Re: Addition with flamethrowers – why game devs don't unit test
#38I don't buy this argument. Most game developers I know have said that unit tests are a waste of time so they never use them, but they're struggling with making changes to utility code and making sure that it doesn't do the wrong thing. Y'know, what unit tests are for. I think the key here is that the perceived cost / benefit ratio is too high. It's the perception that drives their behavior though. I'm in a company no…
> Also, remember that games are not very long-lived pieces of software. You build it, release it, maybe patch it, and move on. This was true a couple decades ago. Nowadays many games are cash cows for decades. Path of Exile was released in 2013, Minecraft in 2011, and World of Warcraft in 2004, and all of those continue to receive regular updates (and have over the course of their lives) and still make plenty of mone…
I still play games that came out a couple decades ago…
Re: Addition with flamethrowers – why game devs don't unit test
#39Arguments against testing tend to fall prey to the von Neumann Objection: they insist there is something tests can’t catch, and then they tell you precisely what it is that tests can’t catch… so you can always imagine writing tests for that specific thing. E.g. this article uses an example of removing the number 5, causing the developer to have to implement a base-9 numbering system. Unit tests that confirm this cust…
Nah, my objection to unit testing is that too often it devolves into what I call "Testing that the code does what the code does." If you find yourself often writing code that also requires updating or rewriting unit tests, your tests are mostly worthless. Unit tests are best for when you have a predefined spec, or you have encountered a specific bug previously and make a test to ensure it doesn't reoccur, or you want to make sure certain weird edge cases are handled correctly. But the obsession with things like 100% unit test coverage is a counterproductive waste of time.
Re: Addition with flamethrowers – why game devs don't unit test
#40Earlier quoted context omitted.
Valve became serious about software quality in Dota 2 around 2017 - about 7 years after launch. Before that game updates were accompanied with lots of bugs that would take weeks to fix. These days, there are still tons of bugs, but much better than before. They just released one of the biggest updates in the game's history this week, and there are hardly any bugs being reported. I am pretty sure there is some sort of…
Reminds me of an article about the testing infrastructure of League and Legends [1] back in 2016. 5500 tests per build in 1 to 2 hours. Games are extremely hard to test. For me it falls into the same category like GUI testing frameworks which imho are extremely annoying and brittle. Except that games are comparable to a user interface consisting of many buttons which you can short and long press and drag around while…
I recall some Minecraft tests being saved worlds with redstone logic that will light a beacon green if it is working or red if not. That's usefull for games like that.
For games like Starcraft 2 with replay functionality, you could probably record/use several matches and test that the behaviour matches the recorded behaviour. If you can make your game have a replay feature you can make use of this, even if you don't ship that replay code.
For things like CYOA type games or decision trees, you could have a logging mechanism that prints out the choices, player stats, hidden stats, etc. and then have a way to run through the decisions, then check the actual log output against the expected output. -- I've done something similar when writing parsers by printing out the parse tree (for AST parser APIs) or the parse events (for reader/SAX parser APIs).
I'm sure there are other techniques for testing other parts of the system. For example, you could test the rendering by saving the render to an image and comparing it against an expected image. IIRC, Firefox does something similar for some systems like the SVG renderer and the HTML paint code.
Various of these features (replay, screenshots) are useful to have in the main game.