Automated Testing for League of Legends

#1

Post by jsnell » Sat, Apr 30, 2016, 11:21 AM UTC

Automated Testing for League of Legends
engineering.riotgames.com

Re: Automated Testing for League of Legends

#2

Post by amenod » Sun, May 01, 2016, 6:35 PM UTC

Nice writeup! Sounds pretty similar to the way serious web apps are being tested (using Selenium & co.). I find it interesting that they built their own testing system though - couldn't they have used some existing framework?

Re: Automated Testing for League of Legends

#3

Post by kyberias » Sun, May 01, 2016, 6:41 PM UTC

Are these all the automated test they run? The article doesn't seem to mention unit tests for example. Do they write and run unit tests? The test class example seems to assert a bunch of stuff that might be easier (and cleaner) to test in a set of unit tests.

Update: In particular this is weird:

> "Tests make use of remote procedure call (RPC) endpoints exposed on the client and the server in order to issue commands and monitor game state. For the most part, tests consist of a fairly linear set of instructions and queries—existing tests cover everything from champion abilities to vision rules to the expected rewards for a minion kill. "

They test "rules for expected rewards" with an out-of-process python test program that connects to the client and server via RPC. Seems unnecessary complex way to test specific game rules.

Is this a symptom of a separate QA team and no developer-written unit-tests?

Re: Automated Testing for League of Legends

#4

Post by swanson » Sun, May 01, 2016, 6:46 PM UTC

A great anecdote in the 'we can't test this code' discussion -- for an aggressive two-week release cycle of this massively popular game, this level of testing has probably paid for itself many times over.

Re: Automated Testing for League of Legends

#5

Post by crdoconnor » Sun, May 01, 2016, 6:57 PM UTC

kyberias wrote: post #3

Are these all the automated test they run? The article doesn't seem to mention unit tests for example. Do they write and run unit tests? The test class example seems to assert a bunch of stuff that might be easier (and cleaner) to test in a set of unit tests. Update: In particular this is weird: > "Tests make use of remote procedure call (RPC) endpoints exposed on the client and the server in order to issue commands…

It's a more loosely coupled way to test than unit tests. That has certain advantages.

Re: Automated Testing for League of Legends

#6

Post by yokohummer7 » Sun, May 01, 2016, 7:03 PM UTC

A release cycle of two weeks looks miracle to an outsider like me. Good to see what's going on internally. I loved the video showing automated champion moves, show me more! Also,

> In Wood 5 we don't use wards anyway, so I see no problem with this critical failure

100% agreed. There's no point in changing to lens because nobody buys a ward!

Re: Automated Testing for League of Legends

#7

Post by badloginagain » Sun, May 01, 2016, 7:07 PM UTC

amenod wrote: post #2

Nice writeup! Sounds pretty similar to the way serious web apps are being tested (using Selenium & co.). I find it interesting that they built their own testing system though - couldn't they have used some existing framework?

A third party solution would be as hard, or harder, than developing an in-house solution. The challenge is setting up the interactions between all gameplay elements- it's going to very unique to each game.

The amount you'd need to abstract to make it reusable for other developers would make it nearly useless.

Re: Automated Testing for League of Legends

#8

Post by cpeterso » Sun, May 01, 2016, 7:11 PM UTC

I'd love to hear more about automated stress testing and LoL's release schedule. Do they have beta releases or code freeze before the two week release dates?

Re: Automated Testing for League of Legends

#9

Post by chipsy » Sun, May 01, 2016, 7:17 PM UTC

kyberias wrote: post #3

Are these all the automated test they run? The article doesn't seem to mention unit tests for example. Do they write and run unit tests? The test class example seems to assert a bunch of stuff that might be easier (and cleaner) to test in a set of unit tests. Update: In particular this is weird: > "Tests make use of remote procedure call (RPC) endpoints exposed on the client and the server in order to issue commands…

No, it's because game engines don't unit test well. There is too much I/O, latency dependent behavior, and shared mutable state to make it feasible outside of some math and low level format or protocol tests. Many game bugs are purely data bugs and result from a misconfiguration or a malformed asset. Manual testing of the result, following a checklist and creatively hammering on the system, is thus the default method.

What Riot is using is a form of integration test where manual user input is emulated to produce a result data set. What you aren't seeing in the test code is "everything else" that was needed to set up a running game state. This technique makes it easy for QA to jump in and see what's happening visually when a failure occurs, eliminating the need to slave away at a checklist.

Re: Automated Testing for League of Legends

#10

Post by finishingmove » Sun, May 01, 2016, 7:19 PM UTC

We see the test checks three things:

  Verifies:
    - KogMaw deals less damage to non-lane minions
    - KogMaw deals percentile magic damage
    - KogMaw deals normal damage to lane minions

and the verify method has three assertions. In my opinion, this should be three tests, each with only one assertion.