Live data from Hacker News

Riot Games: Artificial Latency for Remote Competitors

lolesports.com

111–120 of 160 posts

Re: Riot Games: Artificial Latency for Remote Competitors

#111
post #28

Earlier quoted context omitted.

You could do something like this: • Pre-publish, for each time batch, a public key. You could publish lists of these well in advance. • Let everyone submit, alongside each order, a number arbitrarily selected by them. It does not matter how they select the number, but it would be simpler if everyone chose distinct numbers. • When order processing is done, do it by the order of closeness of the submitted number to the…

Something very similar is done on smart contracts to introduce randomness (at least to the practical extent) on blockchain. Good thought experiment nevertheless to provably inject randomness into a system with untrusting parties.

Oh, there are trust-less coin flip protocols.

Edit: see https://en.wikipedia.org/wiki/Commitment_scheme#Coin_flippin...

Re: Riot Games: Artificial Latency for Remote Competitors

#112
post #68

Distributed consensus is hard. Very hard. Anyone dismissing the problem as “should never have hopped,” hasn’t a clue as to how hard and error prone reliable distributed consensus is. 35 ms lag in overall responsiveness is an eternity in competitive game play. There are videos floating around that show drawing on a tablet surface with various input latencies (perhaps someone has a link, I can’t find them at the moment…

I think you meant https://youtu.be/vOvQCPLkPt4 . It was indeed not too easy to find, due to SEO tactics polluting YT search with gaming-oriented stuff from recent times (where one may actually reach the frame rates necessary for such low latencies in a frame-by-frame display technology).

Re: Riot Games: Artificial Latency for Remote Competitors

#113

A good software lesson here. They built a complex system to tweak tens of ms (at most) of ping to equalize. And they had a bug and it disadvantaged on team. Thus, they have to replay due to unfairness introduced by Riot itself. They could've gone Option 1 - all teams at their natural ping. While it wouldn't be as perfect in theory, it wouldn't have resulted in replaying matches. Plenty of other esports and fighting g…

I think ping differences lead to very interesting effects. One example is, imagine you have a "favor the shooter" system, and you have a 1v1 where one player has a sniper rifle that takes a while to charge to full power when they put it up to their eye to shoot, and you have the ability to become instantly invulnerable when you press the shift key. If the packet that says "she is putting the rifle up to her eye now" arrives 100ms late, then you have a false sense of security, because your mental model of her state of charge is 100ms behind reality. It means you use your invulnerability too late when you think it's time for her to fire a fully-charged shot, and you die instead of live. (This happens in Overwatch; if you play Mei you'll know what it's like to die inside your ice block.)

Another interesting scenario is projectile-based weapons. For reasonable Internet latencies and reasonable in-game distances, the projectile can travel faster than the packet announcing that it has been fired. The game state that gets sent from the shooter over the network is "I fired my weapon, I killed them". At the time you die, your client has none of that information, and since the shooter is favored, you just die out of nowhere. (This is less annoying, because I don't think you could move out of the way fast enough for it to matter. "My opponent's WiFi is bad, so I die half a second later for no reason" still feels pretty bad though. In the intervening milliseconds you had grand plans to win the game!)

All of this, to me, is a great lesson in eventual consistency in distributed systems. People break their keyboards over this stuff. Thinking it will be fine for your more-critical-than-a-casual-video-game system means this stuff will be happening to you multiple times a day, and you can't break your keyboard because you can only blame yourself for brining that system into the world. Tread carefully! Tread very carefully. Shit gets weird when your system has a split brain.

Re: Riot Games: Artificial Latency for Remote Competitors

#114
post #46

Earlier quoted context omitted.

well adjusting the latency is the easy part. doing it in real time while observing the connection and making adjustments, that is when it becomes hard. and they did it basically after the network stack, so if a paket took 40ms they added no latency if it took 20ms they added 15ms and so on. so EACH paket was evaluated and of course this is not an easy feat, because lol is serverside authoritive so the latency must be…

Yea, I guess the question is whether or not to try to do it at the packet level or just keep a running average of ping latency and adjust it over time. They said `adjust the latency (ping) to 35 ms` so I am assuming they are just trying to do something like attempting to maintain an average ping latency of 35 ms.

This is a pretty classic PID here. You're attempting to keep latency at 35ms and based on the latency on each packet as observed by the server, and previously recorded latencies, you can add corrections. With some simulations and tuning I'll bet you can get a pretty good algorithm going.

Simulating this stuff is pretty simple. Model each incoming stream as an MM1 queue and play around with your PID algorithm until it gives you the results you need.

Re: Riot Games: Artificial Latency for Remote Competitors

#115
post #41

Earlier quoted context omitted.

I'd argue the complexity of trying to achieve this at all is too much to bear in practice, especially considering the type of customers this would be inflicted upon. Competitive gamers will now also be wondering if the "fake lag" system is bugged, in addition to all of the other problems that could still exist. Some problems cannot be solved with clever tricks.

if they pulled an IEX and just ran the Busan clients through a few km of fibre to induce delay they could use native pings and verify easily

Yes, but they probably run on AWS and can't use their own hardware.

Re: Riot Games: Artificial Latency for Remote Competitors

#116

Earlier quoted context omitted.

You can in fact have that magical time-travel approach to get apparent latency lower than the real number. Keep a history of the game state at every tick, so you can roll back to any previous instant. If real latency from one client is 50 ms, and you're targeting 35 ms, you roll back the game state by 15 ms each time input comes in from that client, and re-run the game logic from that point forwards and send the resu…

I really dislike games which rewind state based upon latency. It penalizes people with good ping and can lead to very weird game states where a large rollback can cause teleportation of a laggy player as seen by non laggy players. Team Fortress 2 is particularly bad where high latency players can teleport around or you can shoot and hit a laggy player but it doesn’t register by the server due to a rollback. Some play…

There's a difference between backwards reconciliation (rolling the game world back to process a lagged player's input) leading to "shot around a wall", and packet loss/jitter leading to jumpy player and teleporting.

I make this distinction because I actually think backwards reconciliation is table stakes for any kind of precision game and is usually pretty easy to implement. It can also help solve teleporting: you run backwards reconciliation on movement commands also (not just shooting or other actions), and combine with a little player position extrapolation.

Nothing fixes a 700ms, actually I would guess the ceiling on competitive play is somewhere around 200ms with incredible netcode. But 200ms gets you pretty far: across the US or the Atlantic Ocean for sure (packet loss over those distances is another story though).

Re: Riot Games: Artificial Latency for Remote Competitors

#117

They used a complex and buggy solution when a simple solution exists: Use the `tc` traffic control [0] program to configure the Linux kernel to add a fixed amount of latency to traffic from the local players [1]. If the game server does not run on Linux, they could put a Linux bridge/router in front of it. [0] https://en.wikipedia.org/wiki/Tc_%28Linux%29 [1] https://serverfault.com/a/841865

`tc` would give you non-deterministic microbursts that would be hard to quantify as to whether they impacted competitive integrity. This is an incredibly complex problem if they actually care about millisecond level precision in small duty cycles.

Are you saying that Linux's packet delay function is imprecise? Can you link to more information about this?

Re: Riot Games: Artificial Latency for Remote Competitors

#118
post #17

I love the intent and actions behind this post. Finding an unfair bug mid-tournament is serious, and it takes courage and integrity on the part of the technical team to -listen to the players -investigate past what your own analytics are saying -find and fix the bug under time pressure -and disclose the (potentially embarrassing!) initial mistake I think the article itself could have been worded better though. It did…

The technical team acted honorably, but the upper stewardship at Riot games behaved horrendously. They sacrificed the competitive integrity of their sport because they didn't want to exclude RNG from the tournament, because they wanted the audience of the Chinese market. This article goes into more detail, giving a larger context for why this situation shouldn't have never happened in the first place. https://www.inv…

Yeah, the technical details are cool and all but it's unreal how this option was even on the table. It sidesteps one of the biggest reasons for a LAN. What's the point of gathering everyone together if 35ms is going to be added to the ping.

Funnily enough, all this hoohaa was for nothing as the Asian Games got postponed anyway.

Re: Riot Games: Artificial Latency for Remote Competitors

#119

Earlier quoted context omitted.

Yes, the tournament would have been worse without them. What I'm saying is that does not somehow justify penalizing the other teams and the tournament writ large to "even" the playing field if you value competitive integrity. As the article states, increasing the ping doesn't make it more fair for the other teams because they actually traveled to the event. They're physically there. It only makes it more fair for RNG…

> increasing the ping doesn't make it more fair for the other teams because they actually traveled to the event Can't agree. Those teams had an advantage when playing RNG. Taking away that advantage does actually make the event more fair for them, even though they are being disadvantaged.

RNG already have an advantage when they are allowed to play remotely from the comfort of their home. Yes it's not the team's fault they can't travel but it is what it is. There's another tournament in 6 months.

There's a reason this Frankenstein ping is unheard of in bigger esports.

Re: Riot Games: Artificial Latency for Remote Competitors

#120

Earlier quoted context omitted.

You can in fact have that magical time-travel approach to get apparent latency lower than the real number. Keep a history of the game state at every tick, so you can roll back to any previous instant. If real latency from one client is 50 ms, and you're targeting 35 ms, you roll back the game state by 15 ms each time input comes in from that client, and re-run the game logic from that point forwards and send the resu…

I really dislike games which rewind state based upon latency. It penalizes people with good ping and can lead to very weird game states where a large rollback can cause teleportation of a laggy player as seen by non laggy players. Team Fortress 2 is particularly bad where high latency players can teleport around or you can shoot and hit a laggy player but it doesn’t register by the server due to a rollback. Some play…

While the exact logic TF2 uses leaves a lot to be improved, you absolutely need this kind of lag compensation. Without rewinding any amount of latency results in having to aim where an enemy gamer will be by the time your packet reaches the server instead of where they are on your screen. If you remember before they patched the Pyro class you had to aim significantly in front of players because the flames weren't lag-compentated.
Post reply on HN