This is beautiful.
Automated Testing for League of Legends
41–50 of 78 posts
Re: Automated Testing for League of Legends
#42A great anecdote in the 'we can't test this code' discussion -- for an aggressive two-week release cycle of this massively popular game, this level of testing has probably paid for itself many times over.
In this case? Probably. But the idea that catching a bug in production is a huge problem is a myth, pushed by research that was later completely withdrawn.
Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline.
These sort of downtimes hurt customer satisfaction and the bottom line. If serious issues arise often enough, customers lose confidence and patience and may leave your product for a competitor. And while you're offline trying desperately to rush a fix, you're losing revenue.
If the service/product you're offering has competitors, quality matters. Sometimes it's not even the measurable quality, but the perception your customers have. Find me a business owner who thinks major bugs in production are not "a huge problem".
Re: Automated Testing for League of Legends
#43too bad dota 2 is a better game
Funnily enough, the latest patch was rushed out by Valve, and it contained a lot of bugs and strange behavior.
Lots of bugs in Dota 2 seems to be happening because of manual testing.
Re: Automated Testing for League of Legends
#44Are these all the automated test they run? The article doesn't seem to mention unit tests for example. Do they write and run unit tests? The test class example seems to assert a bunch of stuff that might be easier (and cleaner) to test in a set of unit tests. Update: In particular this is weird: > "Tests make use of remote procedure call (RPC) endpoints exposed on the client and the server in order to issue commands…
It's useful as a framework for testing all game interactions. When you think about the amount of unique abilities for the hundred(?) unique characters and how they interact against eachother, the test matrix that results is massive. Additionally, you want to use production server/client code as much has possible, while getting through the tests as quickly as possible (skip front end flow, matchmaking, etc.). Using cl…
>> When you think about the amount of unique abilities for the hundred(?) unique characters and how they interact against eachother, the test matrix that results is massive.
Definitely. They're at 130 champions right now. Each with four spell cast abilities, as well as a passive ability, and unique auto attack mechanics on some of them. There are indeed many interactions with specific abilities between champions. Add in interaction with the map and terrain itself, like unit collision and NPC minions and monsters. Then deal with the 154 items (the current count on the main map) players can add to their champion, many of which also interact with abilities, and some of which introduce additional abilities.
It must be both a nightmare and yet very interesting to handle creating a new champion for the game. Once they get past the step of even deciding how the new champion's abilities should interact with other champions, they then need to code it all and manage to verify that everything works according to expectations. Every once in awhile, there are still bugs with how one ability interacts with another. You can plan it all out, but it has got to be easy to miss something. Too many interactions! :)
Re: Automated Testing for League of Legends
#45A great anecdote in the 'we can't test this code' discussion -- for an aggressive two-week release cycle of this massively popular game, this level of testing has probably paid for itself many times over.
It definitely has. League of Legends is considered a serious e-sport game. Individual tournaments for League of Legends have $2,000,000+ prize pools. If a new bug decided a round of the tournament, it would undermine LOL as a serious contender for this type of attention. There is a terrific business case for investing the time to make this game as stable as possible.
Re: Automated Testing for League of Legends
#46Earlier quoted context omitted.
> But the idea that catching a bug in production is a huge problem is a myth You clearly have never worked in fintech, banks or payments. I personally witnessed the moment we catched a race condition which costed the company n-k euros in missed transactions. And this isn't even considering mission critical software where bugs can (literally) kill.
Relax, I don't think I said bugs are never bad, or devastating. I said the idea that catching a bug in production is a huge problem is a myth. It is not, by default, true. It is true given additional information for a specific case, but is not the general rule.
Bug impacts are highly non-linearly and non-Gaussianly distributed. It doesn't matter if the "average" bug is not that big a deal to find in production, what matters is what the worst thing you can find in production is. Even if the study is not literally correct in the average case, it doesn't take much modeling or much real-world experience to see one is still wise to develop with very similar ideas in mind, if not an even more intense focus on getting bugs identified before production because the paper probably understates the impact given real bug distributions.
(I say "identified" because you don't always fix identified bugs. But you really, really want to have as much as possible identified before production. Production is a terrible place to find bugs for the first time. Yeah, every once in a while an identified bug might have a far worse than expected impact, I've had that happen, but at least in my experience the completely unidentified bugs are what kill you the hardest.)
Re: Automated Testing for League of Legends
#47Earlier quoted context omitted.
Relax, I don't think I said bugs are never bad, or devastating. I said the idea that catching a bug in production is a huge problem is a myth. It is not, by default, true. It is true given additional information for a specific case, but is not the general rule.
Bug impacts are highly non-linearly and non-Gaussianly distributed. It doesn't matter if the "average" bug is not that big a deal to find in production, what matters is what the worst thing you can find in production is. Even if the study is not literally correct in the average case, it doesn't take much modeling or much real-world experience to see one is still wise to develop with very similar ideas in mind, if not…
This is an older style of thinking that is, while safe, slower. Competition who doesn't follow this model and pushes fast, failing often will generally run faster to market than a company who follows this line of thinking. Their software might be buggier, but the concept of "good enough" applies.
If you're building rocket engines that carry people, you think like this. If you're building a social networking website, or a food ordering website, or a home sharing website, the time it takes to go from "oh there's a bug" to "that bug is fixed in production" is a matter of hours, if not minutes.
Re: Automated Testing for League of Legends
#48Earlier quoted context omitted.
In this case? Probably. But the idea that catching a bug in production is a huge problem is a myth, pushed by research that was later completely withdrawn.
Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline. These sort of downtimes hurt customer satisfact…
> Find me a business owner who thinks major bugs in production are not "a huge problem".
"Move fast and break things" - Mark Zuckerberg.
Re: Automated Testing for League of Legends
#49A great anecdote in the 'we can't test this code' discussion -- for an aggressive two-week release cycle of this massively popular game, this level of testing has probably paid for itself many times over.
It definitely has. League of Legends is considered a serious e-sport game. Individual tournaments for League of Legends have $2,000,000+ prize pools. If a new bug decided a round of the tournament, it would undermine LOL as a serious contender for this type of attention. There is a terrific business case for investing the time to make this game as stable as possible.
This has actually happened before at official League of Legends tournaments (that is, a fairly serious bug in a serious match). I believe their policy in the case of a serious bug is to re-do the match, but I am not 100% sure.
Re: Automated Testing for League of Legends
#50Earlier quoted context omitted.
Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline. These sort of downtimes hurt customer satisfact…
> Find me a business owner who thinks major bugs in production are not "a huge problem". "Move fast and break things" - Mark Zuckerberg.
They moved away from that, didn't they?