Addition with flamethrowers – why game devs don't unit test • Hacker News Board

Re: Addition with flamethrowers – why game devs don't unit test

#61

Post by eviks » Sat, May 25, 2024, 5:12 PM UTC

mistercow wrote: post #55

jayd16 wrote: post #23

Earlier quoted context omitted.

So wait, who's being dismissive of who's practices here?

I didn’t say engineers are dismissive of other engineers’ practices. The general pattern is “that makes sense for your field, but we can’t use it because…” followed by silly reasons. I was guilty of this myself back when I was an indie dev. It took me an embarrassingly long time, for example, to admit that git wasn’t just something teams needed to coordinate, and that I should be using it as the sole developer of a p…

Interesting, was the impetus for change some big issue you've run into? Or just a gradual accumulation of knowledge about other people experiences made you reconsider? Or something else?

Re: Addition with flamethrowers – why game devs don't unit test

#62

Post by kbolino » Sat, May 25, 2024, 5:58 PM UTC

wesselbindt wrote: post #60

treflop wrote: post #34

Earlier quoted context omitted.

I’ve seen people slog through untested code where they fear to make a change but I’ve also seen people slog through code with too much test coverage where the tests go through constant churn. I don’t understand why people don’t just add one test even if the codebase otherwise has zero tests if they’re so scared of one area and I don’t get why people keep adding excessive coverage if it’s wasting their time. It’s like…

>too much test coverage where the tests go through constant churn This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details. This has a multitude of possible causes, for example too the tests being too granular (prefer testing at the boundary of your system). I've worked in codebases where test-implementation detail coupling was taken seriously, a…

It feels like there are two levels of test writing proficiency. The first is writing the tests that have high benefit and low cost: e.g. pure functions with comprehensive tabular tests, simple method chains that have well defined sequential behavior and few dependencies, high value regression tests against detailed bug reports, etc. IMO it's harder to argue against writing these tests than to argue for writing them.

Then there's the second level of proficiency, related to what you're discussing with "test-implementation detail coupling". This is the domain of high test coverage, repeatable end-to-end tests, automated QA, etc. I've always struggled with this next level and I've yet to work in any environment where it was done effectively (if at all). It's also harder to argue for this kind of testing because the tests often end up brittle and false negatives drown out the benefits.

Moreover, most of the discourse centers around the first level of proficiency only and it's much harder to find digestible advice for achieving the second.

Re: Addition with flamethrowers – why game devs don't unit test

#63

Post by shaftway » Sat, May 25, 2024, 6:36 PM UTC

Atotalnoob wrote: post #8

shaftway wrote: post #5

Earlier quoted context omitted.

We produce a library that gets included in software made by our clients, and we have several thousand clients. The uptake on new releases is low (most of the clients believe in "if it ain't broke, don't fix it"). So every release has the potential to live in the wild and need support for a long time. We're also in an industry with a ton of competitors. On top of that, the company was founded by some very junior engin…

Thank you for the write up. That seems like a bad scenario with bad technical management. I am wondering if you have considered not trying to implement unit tests and think about end to end tests. This might be easier for antitesting people to buy into because it’s directly ensuring your end users get the desired outcomes. It doesn’t matter what bad terrible practices you have inside your library if the output is cor…

We do have an integration test that runs just before releases. I've never seen it fail, even when something was obviously broken, so I question the utility of it. There's a specific person in charge of maintaining it.

I've opted for option 4: continue to write code the way they want it written and keep cashing my paychecks. In the meantime there are tons of other improvements that I'm working on, some of which have a more direct impact on business revenue (which has a direct impact on my personal revenue).

Re: Addition with flamethrowers – why game devs don't unit test

#64

Post by bluefirebrand » Sat, May 25, 2024, 7:09 PM UTC

wesselbindt wrote: post #60

treflop wrote: post #34

Earlier quoted context omitted.

I’ve seen people slog through untested code where they fear to make a change but I’ve also seen people slog through code with too much test coverage where the tests go through constant churn. I don’t understand why people don’t just add one test even if the codebase otherwise has zero tests if they’re so scared of one area and I don’t get why people keep adding excessive coverage if it’s wasting their time. It’s like…

>too much test coverage where the tests go through constant churn This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details. This has a multitude of possible causes, for example too the tests being too granular (prefer testing at the boundary of your system). I've worked in codebases where test-implementation detail coupling was taken seriously, a…

> This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details

Depending on how high coverage you are aiming for, I find it hard to imagine a way to achieve it without inevitably tying the tests to implementation details

Re: Addition with flamethrowers – why game devs don't unit test

#65

Post by vlovich123 » Sat, May 25, 2024, 7:13 PM UTC

zubspace wrote: post #53

rhdunn wrote: post #40

Earlier quoted context omitted.

I recall some Minecraft tests being saved worlds with redstone logic that will light a beacon green if it is working or red if not. That's usefull for games like that. For games like Starcraft 2 with replay functionality, you could probably record/use several matches and test that the behaviour matches the recorded behaviour. If you can make your game have a replay feature you can make use of this, even if you don't…

You're right about parts, which are mostly state machines. The have a defined input and output. Tests are straightforward to implement and adjust. But recording and replaying matches? Taking screenshots and comparing the output? Just think about it: If you have recorded a match and change the hitpoints of a single creature, the test could possibly fail. And then? Re-record the match? The same applies to screenshots:…

Yes, when you are trying to intentionally change the output, you simply regenerate the gold file to be used as reference (and yes, it should be easy). It’s brittle for sure but it does catch unintentional changes and should be used where relevant (if sparingly). There are definitely existing frameworks that do this (eg Jest calls this snapshot testing and has tooling to make it easy).

I’m sorry your experiences with this kind of stuff have been bad. I’ve generally had good experiences in the machine learning space where we used it judiciously where appropriate but didn’t overdo it.

I don’t see how it can ever hinder you though - you can always choose to go “I don’t care that the output has changed dramaticallly - it’s the new ground truth” as long as you communicate that’s what happening in your commit. What it doesn’t let you do is that the output is different every time you run it but that’s generally a positive (randomness should be intentionally injected deterministically).

Re: Addition with flamethrowers – why game devs don't unit test

#66

Post by armchairhacker » Sat, May 25, 2024, 7:35 PM UTC

wesselbindt wrote: post #60

treflop wrote: post #34

Earlier quoted context omitted.

I’ve seen people slog through untested code where they fear to make a change but I’ve also seen people slog through code with too much test coverage where the tests go through constant churn. I don’t understand why people don’t just add one test even if the codebase otherwise has zero tests if they’re so scared of one area and I don’t get why people keep adding excessive coverage if it’s wasting their time. It’s like…

>too much test coverage where the tests go through constant churn This doesn't sound so much as too much coverage but rather like having your automated tests be coupled to implementation details. This has a multitude of possible causes, for example too the tests being too granular (prefer testing at the boundary of your system). I've worked in codebases where test-implementation detail coupling was taken seriously, a…

Even if the tests aren’t coupled to implementation details, in most projects the specification itself goes through many changes. Furthermore, as the implementation is being changed, it stops depending on some lower-level helper code and requires new code with a different purpose; the tests in the old code turn out to be largely (albeit not entirely) a waste of effort.

Changing specifications and code which turns out to be unnecessary aren’t ideal. but I believe they’re inevitable to some extent (unless the project is a narrow re-implementation of something that already exists). There are questions like “how will people use this product?” and “what will they like/dislike about it?” that are crucial to the specification yet can’t be answered or even predicted very well until there’s already a MVP. And you can’t know exactly what helper classes and functions you will use to implement something until you have the working implementation.

Of course, that doesn’t mean all tests are wasted effort; development will be slower if the developers have to spend more time debugging, due to not knowing where bugs originate from, due to not having tests. There’s a middle ground, where you have tests to catch probable and/or tricky bugs, and tests for code unlikely to be made redundant, but don’t spend too long on unnecessary tests for unnecessary code.

Re: Addition with flamethrowers – why game devs don't unit test

#67

Post by mistercow » Sat, May 25, 2024, 7:40 PM UTC

magoghm wrote: post #59

mistercow wrote: post #56

Earlier quoted context omitted.

I’ve written tests to do exactly that, by adding carefully placed locks that allow the test to control the pace at which each thread advances. It’s not fun but you can do it.

Doesn't inserting locks affect the memory hierarchy consistency mechanisms and therefore interfere with possible race conditions?

That’s not a situation I’ve encountered but “race condition” is an extremely broad category.

Re: Addition with flamethrowers – why game devs don't unit test

#68

Post by mistercow » Sat, May 25, 2024, 7:44 PM UTC

eviks wrote: post #61

mistercow wrote: post #55

Earlier quoted context omitted.

I didn’t say engineers are dismissive of other engineers’ practices. The general pattern is “that makes sense for your field, but we can’t use it because…” followed by silly reasons. I was guilty of this myself back when I was an indie dev. It took me an embarrassingly long time, for example, to admit that git wasn’t just something teams needed to coordinate, and that I should be using it as the sole developer of a p…

Interesting, was the impetus for change some big issue you've run into? Or just a gradual accumulation of knowledge about other people experiences made you reconsider? Or something else?

It was a long time ago, but it was probably having to switch from major feature work to emergency bug fixes that finally became painful enough for me to acknowledge that manual backups weren’t going to cut it.

Re: Addition with flamethrowers – why game devs don't unit test

#69

Post by tmtvl » Sat, May 25, 2024, 9:01 PM UTC

2muchcoffeeman wrote: post #38

feoren wrote: post #4

Earlier quoted context omitted.

> Also, remember that games are not very long-lived pieces of software. You build it, release it, maybe patch it, and move on. This was true a couple decades ago. Nowadays many games are cash cows for decades. Path of Exile was released in 2013, Minecraft in 2011, and World of Warcraft in 2004, and all of those continue to receive regular updates (and have over the course of their lives) and still make plenty of mone…

I still play games that came out a couple decades ago…

Let me guess... Super Metroid? Chrono Trigger? Final Fantasy VI? Ultima Underworld? Symphony of the Night?

There were a few decent games released in the '80s and '90s.

Re: Addition with flamethrowers – why game devs don't unit test

#70

Post by jrockway » Sat, May 25, 2024, 9:21 PM UTC

AndyPa32 wrote: post #49

jrockway wrote: post #18

Bugs are kind of the fun part of games. If every subroutine worked perfectly, you wouldn't have the chaos of real life. Some of players favorite mechanics are just bugs. (Overwatch example: Mercy's super jump, now a legitimate predictable mechanic that everyone can do, not just people that read the forums and watch YouTube videos about bugs. It started out as a bug, and it was so cool and skill-ceiling increasing tha…

> Games are just software engineering. Fun software engineering. I do question the "fun" part. Midnight crunches, unpaid overtime and - as far as I have read - some of the worst working conditions in all of software engineering. I pass.

That is probably true, but you know you suffer for your art and all that. People don't really like software, but they love games. We know that games are just software, but it's so fun, that people forget that. It's pretty cool. Though to me, I kind of like getting 8 hours of sleep a night and playing other people's games. While getting paid more :/