Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Riot Games: Artificial Latency for Remote Competitors

lolesports.com

Post reply on HN
Print view View original

61–70 of 160 posts

Re: Riot Games: Artificial Latency for Remote Competitors

#61

Post by Karrot_Kream »

colinmhayes wrote: post #55
tester756 wrote: post #52

Earlier quoted context omitted.

>The difference was .01 seconds. It's not immediately obvious that there is a problem. in games you really feel ping differences if you were playing e.g for a year on 5ping and then jumped on 30, then you'll feel that in games like LoL and probably CSGO? idk about the second one.

They said the ping was 45 instead of 35. I'm not denying that the pros can tell the difference, I'm saying it wouldn't be obvious to whatever tester they had that it was 10ms off.

As others have said, the pros noticed. Having an incorrect calculation and seeing large jitter spikes can probably push the latency into something like 50ms momentarily, enough for pros to lose their "immersion".

Re: Riot Games: Artificial Latency for Remote Competitors

#62

Post by ear7h »

dematz wrote: post #8

As context this was a big international fiasco. Nobody was acting maliciously but fans of RNG and other teams felt wronged. RNG didn't want to attend due to covid restrictions on travel but Riot really wants Chinese viewers in international tournaments, hence all this online latency work. Pros on other teams were lied to by the false latency information which must be almost paranoia inducing to be told it's only 35ms…

> I feel like that's the one scenario you'd care about, making the Busan players have 35 ping was the whole point

I think the key word is "significantly". The post states that the tool was originally developed for matches where both teams would be remotely connected, so neither team would have a significantly different latency (you can usually enginneer this by the first approach they suggested, of picking a server in between two locations).

Re: Riot Games: Artificial Latency for Remote Competitors

#63

Post by bostonsre »

merb wrote: post #46
bostonsre wrote: post #42

Earlier quoted context omitted.

Think other people have already done the hard part. Wouldn't be too hard to write something that wraps tc and ping. # hping -S -p 1234 somehost.com HPING somehost.com (ens3 1.2.3.4): S set, 40 headers + 0 data bytes len=46 ip=1.2.3.4 ttl=64 DF id=0 sport=1234 flags=SA seq=0 win=26883 rtt=0.9 ms len=46 ip=1.2.3.4 ttl=64 DF id=0 sport=1234 flags=SA seq=1 win=26883 rtt=0.8 ms len=46 ip=1.2.3.4 ttl=64 DF id=0 sport=1234…

well adjusting the latency is the easy part. doing it in real time while observing the connection and making adjustments, that is when it becomes hard. and they did it basically after the network stack, so if a paket took 40ms they added no latency if it took 20ms they added 15ms and so on. so EACH paket was evaluated and of course this is not an easy feat, because lol is serverside authoritive so the latency must be…

Yea, I guess the question is whether or not to try to do it at the packet level or just keep a running average of ping latency and adjust it over time. They said `adjust the latency (ping) to 35 ms` so I am assuming they are just trying to do something like attempting to maintain an average ping latency of 35 ms.

Re: Riot Games: Artificial Latency for Remote Competitors

#64

Post by mleonhard »

jannyfer wrote: post #4

The part I was interested about the most was glossed over. In what situation can this happen? > we realized that there was a calculation error that only manifested in scenarios where the actual ping was significantly lower than the target latency. In this situation the actual latency would be considerably higher than what is displayed on the overlay on the player's screens.

That's also the kind of error that one can easily catch with a unit test.

Re: Riot Games: Artificial Latency for Remote Competitors

#65

Post by jerf »

axg11 wrote: post #24
jedberg wrote: post #20

Earlier quoted context omitted.

It's complex on one end to reduce complexity on the other -- the trading companies wouldn't have to worry about millisecond optimizations if they trading batches were 200ms windows. So the wire lengths wouldn't matter but also not mattering is the processor, memory, software, etc. for the trading companies. Seems like a good tradeoff. And honestly the wire thing probably isn't real. Light moves 30cm in a nanosecond.…

I used to work in HFT. I promise you that companies would still try to exploit randomized batches. There is an advantage to being the very last entrant into a batch (most up to date information). Truly random batches are not trivial to implement and any statistical pattern in the batching could be exploited.

"Truly random batches are not trivial to implement and any statistical pattern in the batching could be exploited."

If an HFT manages to exploit a correctly implemented entropy-pool random number generator using AES-256 to extend the stream as needed, they're welcome to pick up a few more bucks as far as I'm concerned.

Yes, problems have existed before but I'm sure in this case we can assume high-assurance and careful programming, not some fresh grad being assigned the problem and thinking linear congruential generators sound really cool.

Re: Riot Games: Artificial Latency for Remote Competitors

#66

Post by gosukiwi »

tester756 wrote: post #18
gosukiwi wrote: post #14

Earlier quoted context omitted.

There is a huge difference between 0 ping and 35 ping though, particularly in pro-level LoL. It is strategic but that ping can be the difference between flashing a skillshot or not, and swinging the whole course of a teamfight. Of course 35 ping is not THAT bad and I agree, while unfair, it wouldnt be the end of the world, it's not even Worlds, just MSI, an invitational event.

MSI is "just" 50% of international tournaments imo it's very serious tournament

It is a very serious tournament, it's just not as important as Worlds

Re: Riot Games: Artificial Latency for Remote Competitors

#67

Post by mleonhard »

They used a complex and buggy solution when a simple solution exists: Use the `tc` traffic control [0] program to configure the Linux kernel to add a fixed amount of latency to traffic from the local players [1]. If the game server does not run on Linux, they could put a Linux bridge/router in front of it.

[0] https://en.wikipedia.org/wiki/Tc_%28Linux%29

[1] https://serverfault.com/a/841865

Re: Riot Games: Artificial Latency for Remote Competitors

#68

Post by joshuak »

Distributed consensus is hard. Very hard. Anyone dismissing the problem as “should never have hopped,” hasn’t a clue as to how hard and error prone reliable distributed consensus is. 35 ms lag in overall responsiveness is an eternity in competitive game play.

There are videos floating around that show drawing on a tablet surface with various input latencies (perhaps someone has a link, I can’t find them at the moment). 35ms latency is very noticeable to anyone never mind professional competitors.

Because of the tricks game designers must pull off in lieu of proper distributed consensus which has hard requirements bound by the laws of physics, it is completely likely there are lots of bugs in the system. I think Riot did the best anyone could reasonably have expected of them, and the write up is particularly informative and helpful.

Re: Riot Games: Artificial Latency for Remote Competitors

#69

Post by [deleted] »

vikingerik wrote: post #44
dematz wrote: post #8

As context this was a big international fiasco. Nobody was acting maliciously but fans of RNG and other teams felt wronged. RNG didn't want to attend due to covid restrictions on travel but Riot really wants Chinese viewers in international tournaments, hence all this online latency work. Pros on other teams were lied to by the false latency information which must be almost paranoia inducing to be told it's only 35ms…

You can in fact have that magical time-travel approach to get apparent latency lower than the real number. Keep a history of the game state at every tick, so you can roll back to any previous instant. If real latency from one client is 50 ms, and you're targeting 35 ms, you roll back the game state by 15 ms each time input comes in from that client, and re-run the game logic from that point forwards and send the resu…

[deleted]

Re: Riot Games: Artificial Latency for Remote Competitors

#70

Post by upwardbound »

bobogei81123 wrote: post #38
jannyfer wrote: post #4

The part I was interested about the most was glossed over. In what situation can this happen? > we realized that there was a calculation error that only manifested in scenarios where the actual ping was significantly lower than the target latency. In this situation the actual latency would be considerably higher than what is displayed on the overlay on the player's screens.

My guess is that they did not add in the client delay.

Exactly right.

They added too much latency because they forgot to account for the latency they added in the client. You can see from their architecture diagram (Figure 4) that the latency measurement didn't include the client delay.

https://images.contentstack.io/v3/assets/bltad9188aa9a70543a...

The blog post states: "The existing network monitoring system measured the latency at the networking layer as shown by the green arrow."

Post reply on HN