Automated Testing for League of Legends • Hacker News Board

Re: Automated Testing for League of Legends

#11

Post by badloginagain » Sun, May 01, 2016, 7:22 PM UTC

kyberias wrote: post #3

Are these all the automated test they run? The article doesn't seem to mention unit tests for example. Do they write and run unit tests? The test class example seems to assert a bunch of stuff that might be easier (and cleaner) to test in a set of unit tests. Update: In particular this is weird: > "Tests make use of remote procedure call (RPC) endpoints exposed on the client and the server in order to issue commands…

It's useful as a framework for testing all game interactions. When you think about the amount of unique abilities for the hundred(?) unique characters and how they interact against eachother, the test matrix that results is massive.

Additionally, you want to use production server/client code as much has possible, while getting through the tests as quickly as possible (skip front end flow, matchmaking, etc.).

Using client/server endpoint calls are great because it allows these integration tests to be a list of instructions (the same instructions used by production code) to create a very deep test suite.

Integration testing is critical for games, where you have so many independent units interacting and changing each other's states.

Re: Automated Testing for League of Legends

#12

Post by derFunk » Sun, May 01, 2016, 7:23 PM UTC

Wow, this is pretty sophisticated and commendable. I'd love to have the resources to do something similar. In contrast to 'standard' application development, automated testing is really rare in the non-AAA games industry. At least in terms of logic-/active gameplay testing. It's pure luxury, you can only do it if you can afford it. In a project-based work for hire game shop this is an almost unthinkable thing to do, because you don't get it sold to your contractors/customers. They just won't pay for the effort you don't our directly into the game. The only thing you can do is to develop your own automated testing framework over time and over projects, which is a tedious thing to do because you cannot really focus on it (because it's not a first class citizen in your project schedule).

Re: Automated Testing for League of Legends

#13

Post by badloginagain » Sun, May 01, 2016, 7:26 PM UTC

cpeterso wrote: post #8

I'd love to hear more about automated stress testing and LoL's release schedule. Do they have beta releases or code freeze before the two week release dates?

If using an svn/p4 source control method, they probably branch mainline into a release branch, test, then branch to a staging, test against production data, etc, all the while fixing any necessary bugs. This allows development work to continue on mainline while keeping the release testing continuous and stable.

Re: Automated Testing for League of Legends

#14

Post by badloginagain » Sun, May 01, 2016, 7:29 PM UTC

derFunk wrote: post #12

Wow, this is pretty sophisticated and commendable. I'd love to have the resources to do something similar. In contrast to 'standard' application development, automated testing is really rare in the non-AAA games industry. At least in terms of logic-/active gameplay testing. It's pure luxury, you can only do it if you can afford it. In a project-based work for hire game shop this is an almost unthinkable thing to do,…

I hope it started as a much more humble tool, that kept growing in size as the company did. Each release it gets a bit more useful, until you hit that critical mass point where's it's a pillar of development.

Re: Automated Testing for League of Legends

#15

Post by swanson » Sun, May 01, 2016, 7:34 PM UTC

finishingmove wrote: post #10

We see the test checks three things: Verifies: - KogMaw deals less damage to non-lane minions - KogMaw deals percentile magic damage - KogMaw deals normal damage to lane minions and the verify method has three assertions. In my opinion, this should be three tests, each with only one assertion.

I tend to agree, but sometimes in these kind of instrumented test environments, the setup cost of each test case might be very high so maybe the team found it reasonable to group all the assertions into a single test (especially given that the suite is already taking 1-2 hours to give feedback to developers).

Re: Automated Testing for League of Legends

#16

Post by badloginagain » Sun, May 01, 2016, 7:37 PM UTC

swanson wrote: post #15

finishingmove wrote: post #10

We see the test checks three things: Verifies: - KogMaw deals less damage to non-lane minions - KogMaw deals percentile magic damage - KogMaw deals normal damage to lane minions and the verify method has three assertions. In my opinion, this should be three tests, each with only one assertion.

I tend to agree, but sometimes in these kind of instrumented test environments, the setup cost of each test case might be very high so maybe the team found it reasonable to group all the assertions into a single test (especially given that the suite is already taking 1-2 hours to give feedback to developers).

It appears they're grouping by ability.

Re: Automated Testing for League of Legends

#17

Post by andrepd » Sun, May 01, 2016, 7:40 PM UTC

derFunk wrote: post #12

Wow, this is pretty sophisticated and commendable. I'd love to have the resources to do something similar. In contrast to 'standard' application development, automated testing is really rare in the non-AAA games industry. At least in terms of logic-/active gameplay testing. It's pure luxury, you can only do it if you can afford it. In a project-based work for hire game shop this is an almost unthinkable thing to do,…

And yet they can't rewrite their ridiculously inefficient launcher properly :^)

Re: Automated Testing for League of Legends

#18

Post by huac » Sun, May 01, 2016, 8:18 PM UTC

cpeterso wrote: post #8

I'd love to hear more about automated stress testing and LoL's release schedule. Do they have beta releases or code freeze before the two week release dates?

They have a public beta environment that is set up for feature testing, rather than stress testing. The PBE does see code rollbacks (if features are incomplete) before releases, so I believe the game logic on the PBE represents basically what will be moved to the live server.

Re: Automated Testing for League of Legends

#19

Post by quaunaut » Sun, May 01, 2016, 8:20 PM UTC

andrepd wrote: post #17

derFunk wrote: post #12

Wow, this is pretty sophisticated and commendable. I'd love to have the resources to do something similar. In contrast to 'standard' application development, automated testing is really rare in the non-AAA games industry. At least in terms of logic-/active gameplay testing. It's pure luxury, you can only do it if you can afford it. In a project-based work for hire game shop this is an almost unthinkable thing to do,…

And yet they can't rewrite their ridiculously inefficient launcher properly :^)

Actually, the launcher was replaced under 2 years ago. The matchmaking client has entered alpha, and is accompanied with rewrites to a lot of Riot systems to help it all work together.

Re: Automated Testing for League of Legends

#20

Post by finishingmove » Sun, May 01, 2016, 8:20 PM UTC

swanson wrote: post #15

finishingmove wrote: post #10

We see the test checks three things: Verifies: - KogMaw deals less damage to non-lane minions - KogMaw deals percentile magic damage - KogMaw deals normal damage to lane minions and the verify method has three assertions. In my opinion, this should be three tests, each with only one assertion.

I tend to agree, but sometimes in these kind of instrumented test environments, the setup cost of each test case might be very high so maybe the team found it reasonable to group all the assertions into a single test (especially given that the suite is already taking 1-2 hours to give feedback to developers).

I am well aware of that and I agree on being pragmatic, but more often than not, the reason for multiple assertions per test is developer laziness and/or unawareness of the benefits of isolating failures by having just one assertion per test.