Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Automated Testing for League of Legends

engineering.riotgames.com

Post reply on HN
Print view View original

71–78 of 78 posts

Re: Automated Testing for League of Legends

#71

Post by MichaelBurge »

dimino wrote: post #34
debaserab2 wrote: post #30

Earlier quoted context omitted.

When customers discover bugs before you do, it reflects poorly on your reputation and harms the trust you have established with the consumer of your application. That's not a myth.

Depends on your domain. Facebook bugs are less costly to deploy fixes for than router firmware bugs. The specific thing I referred to as a myth, though? That is completely fabricated, the guy lied in his research. Total myth.

I could publish a study saying smoking is bad for you, intentionally make some mistakes, and have the study withdrawn using strong words like 'fabricated' or 'lied'.

The result would be a dimino-compliant way of establishing that smoking is not bad for you. Or at least, that "smoking is bad for you" is a myth.

Of course, for smoking we have dozens of studies and a strong understanding of the mechanisms, so this trick wouldn't fool anyone. But, just be careful about letting fraudsters determine your opinion on a topic, whether for or against, whether they're caught or not.

As for bugs specifically, there are huge benefits to being able to reason about systems as if they are bug-free. It's a PITA when your libraries, kernel, compiler, upstream API, etc. don't behave as expected. As professional software developers, we can work around them; users can sometimes get really confused when things don't work as expected. Sometimes they're scared to tell you.

Smaller companies can probably bear more bugs, since they can give individual users more attention. Larger ones tend to need to work more reliably.

Re: Automated Testing for League of Legends

#72

Post by JabavuAdams »

jnardiello wrote: post #36
dimino wrote: post #28

Earlier quoted context omitted.

In this case? Probably. But the idea that catching a bug in production is a huge problem is a myth, pushed by research that was later completely withdrawn.

> But the idea that catching a bug in production is a huge problem is a myth You clearly have never worked in fintech, banks or payments. I personally witnessed the moment we catched a race condition which costed the company n-k euros in missed transactions. And this isn't even considering mission critical software where bugs can (literally) kill.

So, excluding life-critical software, it's still a business decision.

Engineering will naturally tend to geek out on infrastructure projects like this and they need to be kept focused on the business case. There has to be some push-back.

What is the cost of even one engineer working full-time on build verification?

The thing you really want to avoid is getting blind-sided by devastating bugs or systemic process problems. So it's a balance.

I've just seen many cases where projects got bogged down as developers built their super-uber-build-test framework. It's easy for people to push these projects through without rationally investigating the cost / benefit, because ... because ... you're not seriously suggesting that we shouldn't test more, you monster!?

Re: Automated Testing for League of Legends

#73

Post by jerf »

dimino wrote: post #47
jerf wrote: post #46

Earlier quoted context omitted.

Bug impacts are highly non-linearly and non-Gaussianly distributed. It doesn't matter if the "average" bug is not that big a deal to find in production, what matters is what the worst thing you can find in production is. Even if the study is not literally correct in the average case, it doesn't take much modeling or much real-world experience to see one is still wise to develop with very similar ideas in mind, if not…

This is an older style of thinking that is, while safe, slower. Competition who doesn't follow this model and pushes fast, failing often will generally run faster to market than a company who follows this line of thinking. Their software might be buggier, but the concept of "good enough" applies. If you're building rocket engines that carry people, you think like this. If you're building a social networking website,…

The problem with bugs is not the bugs. It's what the bugs do. Bork up your database and the bug fix is not "minutes". (Especially if you don't notice for a while.) Piss off your customers and the bug fix is not "minutes". Screw up your money handling and the fix is not "minutes". (Well, your part of the fix may be just one minute, though that minute will be "you're fired".)

If you've only ever encountered bugs that can be fixed in "minutes", you're either very lucky, or not working on anything all that important. Or possibly, not very perceptive and you've actually got a mess on your hands and you haven't realized it yet. I've inherited systems run by such people; they think everything's hunky dory but it turns out you can hardly figure out how to connect two records in the database together correctly because they were too smart for academic bullshit like referential integrity, and, lo, their database was low on the integrity. Tends to work, except the amount of elbow grease required increases without bound until it exceeds the capabilities of the developer in question, and suddenly, one day, they wake up and their job is in serious jeopardy because of what's going down and what they can't get back up.

You assert that this is the "older" style of thinking, which shows a lack of understanding of your computer programming history. What you advocate is "cowboy" thinking, and it predates what I'm discussing by quite a bit. The entire 1970s was basically run this way, and it shows in those handful of remaining technologies that are still with us, and their tendency to just pound through problems as if errors are an admission of failure.

Re: Automated Testing for League of Legends

#74

Post by badloginagain »

developer2 wrote: post #44
badloginagain wrote: post #11

Earlier quoted context omitted.

It's useful as a framework for testing all game interactions. When you think about the amount of unique abilities for the hundred(?) unique characters and how they interact against eachother, the test matrix that results is massive. Additionally, you want to use production server/client code as much has possible, while getting through the tests as quickly as possible (skip front end flow, matchmaking, etc.). Using cl…

>> When you think about the amount of unique abilities for the hundred(?) unique characters and how they interact against eachother, the test matrix that results is massive. Definitely. They're at 130 champions right now. Each with four spell cast abilities, as well as a passive ability, and unique auto attack mechanics on some of them. There are indeed many interactions with specific abilities between champions. Add…

That's why the BVS they've built becomes so important. The amount of test-cases you have to satisfy go beyond the human mind's ability to hold.

But if you have a farm that tests v1 of your new champs abilities against all others, you know exactly where the 20% of problems will lie. This is where designers earn their paycheck- two abilities that won't resolve by design, and where the programmer puts 80% of their work.

80/20 rule scales with the product, I suppose :)

Re: Automated Testing for League of Legends

#75

Post by dimino »

MichaelBurge wrote: post #71
dimino wrote: post #34

Earlier quoted context omitted.

Depends on your domain. Facebook bugs are less costly to deploy fixes for than router firmware bugs. The specific thing I referred to as a myth, though? That is completely fabricated, the guy lied in his research. Total myth.

I could publish a study saying smoking is bad for you, intentionally make some mistakes, and have the study withdrawn using strong words like 'fabricated' or 'lied'. The result would be a dimino-compliant way of establishing that smoking is not bad for you. Or at least, that "smoking is bad for you" is a myth. Of course, for smoking we have dozens of studies and a strong understanding of the mechanisms, so this trick…

I don't think the characterization you're making of my argument is accurate, and I don't get the sense you're interested in understanding what I'm explaining here, so I'm probably done.

I'll leave you with a recommendation to read The Startup Owner's Handbook, and listen to the Stanford lectures given by Sam Altman and friends, who talk a lot more about this.

Re: Automated Testing for League of Legends

#76

Post by dimino »

jerf wrote: post #73
dimino wrote: post #47

Earlier quoted context omitted.

This is an older style of thinking that is, while safe, slower. Competition who doesn't follow this model and pushes fast, failing often will generally run faster to market than a company who follows this line of thinking. Their software might be buggier, but the concept of "good enough" applies. If you're building rocket engines that carry people, you think like this. If you're building a social networking website,…

The problem with bugs is not the bugs. It's what the bugs do. Bork up your database and the bug fix is not "minutes". (Especially if you don't notice for a while.) Piss off your customers and the bug fix is not "minutes". Screw up your money handling and the fix is not "minutes". (Well, your part of the fix may be just one minute, though that minute will be "you're fired".) If you've only ever encountered bugs that c…

I think you're having a reaction to the higher risk profile that comes with breaking in production. That's understandable, but wrongheaded. Plenty of companies are very successful with the mindset as I have described.

Netflix runs their Simian Army in production, for example. They sometimes pull infrastructure pieces offline intentionally to test their resilience. A good production infrastructure is incredibly resilient.

Re: Automated Testing for League of Legends

#77

Post by JamieAli »

developer2 wrote: post #42
dimino wrote: post #28

Earlier quoted context omitted.

In this case? Probably. But the idea that catching a bug in production is a huge problem is a myth, pushed by research that was later completely withdrawn.

Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline. These sort of downtimes hurt customer satisfact…

I'll look into this and let you know what's going on, mate. I've been playing League of Legends for a couple of months now but i'm still really bad at the game. Enjoying the game so much though, just hopefully I can get better at the game soon.

I'm definitely improving now that I'm using this: http://gamesinfun.com/league-of-legends

Re: Automated Testing for League of Legends

#78

Post by Too »

dimino wrote: post #48
developer2 wrote: post #42

Earlier quoted context omitted.

Depends on the bug. Sticking to League of Legends as an example, is it really the worst thing if they have to disable a champion for a day or two to fix a bug? Not really. It still happens fairly often, and the impact is inconvenient but it's not the end of the world. Now how about a bug that prevents logins or crashes the backend? This completely take the game offline. These sort of downtimes hurt customer satisfact…

> Find me a business owner who thinks major bugs in production are not "a huge problem". "Move fast and break things" - Mark Zuckerberg.

Break things does not imply "break things in production".
Post reply on HN