Live data from Hacker News

Addition with flamethrowers – why game devs don't unit test

pixelatedplaygrounds.com

71–80 of 96 posts

Re: Addition with flamethrowers – why game devs don't unit test

#71

Earlier quoted context omitted.

Reminds me of an article about the testing infrastructure of League and Legends [1] back in 2016. 5500 tests per build in 1 to 2 hours. Games are extremely hard to test. For me it falls into the same category like GUI testing frameworks which imho are extremely annoying and brittle. Except that games are comparable to a user interface consisting of many buttons which you can short and long press and drag around while…

I doubt Dota 2 devs are writing code like this to test. The game is far too complicated, even more so than league, and changes a lot over the years, for this to be viable. Dota 2 and openai had a collaboration in 2018ish, and during this time the Dota 2 bots system was reworked completely. They already can generate videos of every spell in action [1], and I would assume this is done by asking AI bots to demonstrate t…

> They already can generate videos of every spell in action [1]

I'm fairly certain those videos are all handmade. (Yes, all 500+ of them.) Notice that the videos for each hero are recorded in different locations on the map, and the "victim" hero isn't always the same.

Re: Addition with flamethrowers – why game devs don't unit test

#72
I don't see why design needs to have so much impact on whether there are Unit Tests. Most unit tests should be much lower level than anything which would be balanced or user tested out of relevance. You want stuff like "can the player character jump and clear these sets of obstacles which we are building all of our levels with?", and then a nice script that takes prerecorded input and sees if the outcome is determinant. So, this way, if someone inadvertantly changes gravity, or friction, or whatever in the basic systems that determine locomotion, you'll catch it early.

Now, do most game makers take the time to do this? No, because they will likely have a lot else to do and make an excuse not to do it. However, for the most vital tech foundations, it is a good idea.

What gamedev does tend to do more often is smoke testing. Just load up each level and see if it runs out of memory or not. Automated testing on build to see if something broke. It's less granular than unit testing, but when you're building over and over in the heat of a closeout on a project, this type of thing can tease out a breaking bug early as well.

Overall, I like the title of the OP article, but not much that's said within.

Re: Addition with flamethrowers – why game devs don't unit test

#73
post #28
post #9

Arguments against testing tend to fall prey to the von Neumann Objection: they insist there is something tests can’t catch, and then they tell you precisely what it is that tests can’t catch… so you can always imagine writing tests for that specific thing. E.g. this article uses an example of removing the number 5, causing the developer to have to implement a base-9 numbering system. Unit tests that confirm this cust…

Tests can't catch race conditions in multithreaded code. Now that I told you what the tests can't catch, can you imagine writing tests for that specific thing?

> Tests can't catch race conditions in multithreaded code.

Citation needed.

> can you imagine

Yes I can, because several languages have tooling built specifically for finding those race conditions.

If you built it, you can test it. If you can’t test it, you don’t understand what you built.

Re: Addition with flamethrowers – why game devs don't unit test

#74

Earlier quoted context omitted.

>too much test coverage where the tests go through constant churn This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details. This has a multitude of possible causes, for example too the tests being too granular (prefer testing at the boundary of your system). I've worked in codebases where test-implementation detail coupling was taken seriously, a…

> This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details Depending on how high coverage you are aiming for, I find it hard to imagine a way to achieve it without inevitably tying the tests to implementation details

Decoupling tests from what they test does require a concerted effort, and is a skill that requires practice, but in general not as much as you'd think. Most devs are quite comfortable getting rid of coupling between two non-test components (functions, classes, services, whatever) of their system. The main mental hurdle seems to be treating your automated test code as any other code, capable and worthy of being decoupled.

There will always be some coupling between test and testee (for example, if I change my `double` function from doing `x -> 2 * x` to `x -> (x, x)`, my tests for `double` better fail), but there is a lot of coupling which is unnecessary, and this can often be removed. Like decoupling any two pieces of code, there is no one size fits all solution. There are some common sources of unnecessary coupling, and some rules of thumb to avoid them.

For example, let's say we have a (public) sorting routine `sort` which consists of two (private) phases: `prep` and `finish`. The implementation of `sort` would look like

  function sort(xs):
   prepped_xs = prep(xs)
   return finish(prepped_xs)
Let's look at two ways to write a test suite for `sort`.

The first is quite simple, just chuck in a bunch of unsorted arrays of varying sizes and degrees of unsortedness, and assert that what comes out is sorted:

  assert sort([]) == []
  assert sort([3, 2, 1]) == [1, 2, 3]
  etc
The other way is to make the tests more specific by testing `sort`, `prep`, and `finish` separately. For example, we might mock `prep` and do

  x = [1, 2]
  sort(x)
  mocked_prep.assert_called_with(x)
  // and something similar for finish
and have individual tests for `prep` and `finish`:

  assert is_prepped(prep([1, 2]))
  etc
  assert is_sorted(finish(prepped_xs))
  etc
Now suppose you decide that the code would be a lot more readable if some functionality of `prep` would move to `finish`. Nothing changes in the functionality of `sort` as this is purely a refactoring. When you make this change, the first test suite will pass, but the second will fail and require changes (i.e., it is coupled to implementation details of `sort`), because you've changed what `prep` and `finish` do.

So here, by increasing the scope of your test suite from `prep` and `finish` to `sort`, you've achieved looser coupling between the test suite and what is being tested. This can be applied much more generally: by testing at the boundary of your system (at a higher scope), you achieve looser coupling between your tests and your code. That is, you'll have fewer false failures.

Re: Addition with flamethrowers – why game devs don't unit test

#75

Earlier quoted context omitted.

Valve became serious about software quality in Dota 2 around 2017 - about 7 years after launch. Before that game updates were accompanied with lots of bugs that would take weeks to fix. These days, there are still tons of bugs, but much better than before. They just released one of the biggest updates in the game's history this week, and there are hardly any bugs being reported. I am pretty sure there is some sort of…

Reminds me of an article about the testing infrastructure of League and Legends [1] back in 2016. 5500 tests per build in 1 to 2 hours. Games are extremely hard to test. For me it falls into the same category like GUI testing frameworks which imho are extremely annoying and brittle. Except that games are comparable to a user interface consisting of many buttons which you can short and long press and drag around while…

In my experience (full stack web development), unit tests are mostly useless and it is the high-level system tests which add real value. Unfortunately it can take a fair amount of work or skill to architect the test suite in the first place, but once it’s working you can write elegant tests that verify large swathes of code with fairly few lines of test code.

I think UI testing in general is hard though, and given how large a part of games involves UI, that’d be the real reason games don’t have much tests.

Re: Addition with flamethrowers – why game devs don't unit test

#76
post #34
post #2

I don't buy this argument. Most game developers I know have said that unit tests are a waste of time so they never use them, but they're struggling with making changes to utility code and making sure that it doesn't do the wrong thing. Y'know, what unit tests are for. I think the key here is that the perceived cost / benefit ratio is too high. It's the perception that drives their behavior though. I'm in a company no…

I’ve seen people slog through untested code where they fear to make a change but I’ve also seen people slog through code with too much test coverage where the tests go through constant churn. I don’t understand why people don’t just add one test even if the codebase otherwise has zero tests if they’re so scared of one area and I don’t get why people keep adding excessive coverage if it’s wasting their time. It’s like…

Overtesting usually comes from TDD cargo culting where literally every function in the code, no matter how small, is a "unit" that needs to be tested.

Re: Addition with flamethrowers – why game devs don't unit test

#77

Earlier quoted context omitted.

Reminds me of an article about the testing infrastructure of League and Legends [1] back in 2016. 5500 tests per build in 1 to 2 hours. Games are extremely hard to test. For me it falls into the same category like GUI testing frameworks which imho are extremely annoying and brittle. Except that games are comparable to a user interface consisting of many buttons which you can short and long press and drag around while…

In my experience (full stack web development), unit tests are mostly useless and it is the high-level system tests which add real value. Unfortunately it can take a fair amount of work or skill to architect the test suite in the first place, but once it’s working you can write elegant tests that verify large swathes of code with fairly few lines of test code. I think UI testing in general is hard though, and given ho…

Part of it is terminology. You get "unit testing" and "functional testing" and "integration testing" and "system testing" thrown around, often with people meaning different things by these, and vague definitions that partially or wholly overlap.

My rule of thumb is really simple: a test should always be defined in terms of what the user expects. Thus for most apps you should, at the minimum, have tests corresponding to their functional specification. In addition, if the app contains functionality that is consistently reused inside (i.e. embedded libraries), then users of those libraries are the code that calls into them, and so there should also be tests at that boundary (but only after you wrote the high-level tests). Repeat recursively until you get to the bottom.

Re: Addition with flamethrowers – why game devs don't unit test

#78
post #19
post #16

Earlier quoted context omitted.

How do you stay sane working with clowns?

Many of the things he just described are a rational response to historical circumstance. It's fine to say "we're in a bad place" but that's not the same as saying "we're currently making bad decisions".

The inevitable end result of this approach though is that at some point they will be unable to ship new releases that meet the quality bar of their clients.

Re: Addition with flamethrowers – why game devs don't unit test

#79

Earlier quoted context omitted.

I think Minecraft was originally written in Java and rewritten in a good programming language (i.e. not Java).

Being written in Java was probably instrumental in enabling the huge modding community around Minecraft. Which in turn was probably in large part responsible for its success.

And more to the point for this thread, writing it in Java let Notch build and iterate extremely quickly. Minecraft originally came out of a 24 hour game writing competition in which most competitors were using C++, but Notch always used Java because coding speed was the most critical thing in that context.

Re: Addition with flamethrowers – why game devs don't unit test

#80

Earlier quoted context omitted.

Reminds me of an article about the testing infrastructure of League and Legends [1] back in 2016. 5500 tests per build in 1 to 2 hours. Games are extremely hard to test. For me it falls into the same category like GUI testing frameworks which imho are extremely annoying and brittle. Except that games are comparable to a user interface consisting of many buttons which you can short and long press and drag around while…

In my experience (full stack web development), unit tests are mostly useless and it is the high-level system tests which add real value. Unfortunately it can take a fair amount of work or skill to architect the test suite in the first place, but once it’s working you can write elegant tests that verify large swathes of code with fairly few lines of test code. I think UI testing in general is hard though, and given ho…

Agreed. We don't have a lot of UI unit tests in our software in our day job (almost none), but we have extensive tests for utility and data processing functions.

And that's pretty much the same for me in the game I'm making in my spare time. I have no unit tests for UI (it's not worth it, I can easily see when something in the UI isn't working, it's more important for me to just log the bug so I don't forget about it).

But for game logic, like verifying calculations for A.I. are happening as expected, or functions that manipulate numbers on the screen in different ways (scores, power adjustment, etc), yeah I write unit tests for those. And to the article's point, sometimes I have to significantly adjust or redo or even scrap them because I happened to think of a different mechanism and it seems to play better.

There was a long time (over about a dozen games I made) where I never bothered to write a unit test, and I still might not for a tiny game. But for my most recent bigger game (which I started a few years ago), I finally decided to write a few for some tricky numeric logic in the game, and it immediately helped me resolve a logic bug I was seeing periodically but was having a hard time pinning down the cause of it with breakpoints and logs. So I do try to do it more often for checking those things.

Post reply on HN