Riot Games: Artificial Latency for Remote Competitors • Hacker News Board

Re: Riot Games: Artificial Latency for Remote Competitors

#91

Post by [deleted] » Tue, May 17, 2022, 9:30 PM UTC

chabons wrote: post #85

someweirdperson wrote: post #51

Artificial or natural delays, I am somewhat confused that one end measuring/calculating ping can end up with a wrong value. Transmit at local time x, receive at local time y, subtract. It only needs a working local clock. Second surprising thing is artifical latency added on the client. Anything on the client I would avoid, if possible. Third, this all looks like magically connecting two ends. But it is packets flowi…

I think the symmetric delay here makes sense, since you're trying to simulate the latency of a (likely) symmetric connection between another client and the server. If you only add latency on the transmission, then 2 actions taken at the same time by players with different amounts of physical latency will be processed at different times. If we assume the game clocks on each client and the server are in sync, then this…

[deleted]

Re: Riot Games: Artificial Latency for Remote Competitors

#92

Post by jannyfer » Tue, May 17, 2022, 9:34 PM UTC

upwardbound wrote: post #70

bobogei81123 wrote: post #38

Earlier quoted context omitted.

My guess is that they did not add in the client delay.

Exactly right. They added too much latency because they forgot to account for the latency they added in the client. You can see from their architecture diagram (Figure 4) that the latency measurement didn't include the client delay. https://images.contentstack.io/v3/assets/bltad9188aa9a70543a... The blog post states: "The existing network monitoring system measured the latency at the networking layer as shown by the…

That kind of error would apply all the time.

But the blog post states: "a calculation error that only manifested in scenarios where the actual ping was significantly lower than the target latency".

Re: Riot Games: Artificial Latency for Remote Competitors

#93

Post by reitzensteinm » Tue, May 17, 2022, 9:38 PM UTC

skocznymroczny wrote: post #58

jchw wrote: post #56

Earlier quoted context omitted.

This strategy is commonly referred to as being in the class of “latency hiding” as it doesn’t actually improve latency, it just allows you to reduce apparent latency. In practice it’s still useful but I think it’s mainly useful because it allows for local input to be buffered significantly less than if you were to compensate for lag by adding buffering only. However, that is not always useful. It is mainly useful in…

LoL is not using lockstep. Lockstep has two big disadvantages for a competitive game. Tt requires both client to have the full game state to run the simulation, which makes creation of maphacks trivial. The second issue with lockstep is that if one player is lagging, all players are lagging, which is also not something you'd like to have in a fast paced competitive game.

What to do when one player is lagging is a design choice. You can instead keep the game going and feed in their delayed inputs.

Re: Riot Games: Artificial Latency for Remote Competitors

#94

Post by [deleted] » Tue, May 17, 2022, 9:44 PM UTC

chabons wrote: post #85

someweirdperson wrote: post #51

Artificial or natural delays, I am somewhat confused that one end measuring/calculating ping can end up with a wrong value. Transmit at local time x, receive at local time y, subtract. It only needs a working local clock. Second surprising thing is artifical latency added on the client. Anything on the client I would avoid, if possible. Third, this all looks like magically connecting two ends. But it is packets flowi…

I think the symmetric delay here makes sense, since you're trying to simulate the latency of a (likely) symmetric connection between another client and the server. If you only add latency on the transmission, then 2 actions taken at the same time by players with different amounts of physical latency will be processed at different times. If we assume the game clocks on each client and the server are in sync, then this…

[deleted]

Re: Riot Games: Artificial Latency for Remote Competitors

#95

Post by upwardbound » Tue, May 17, 2022, 9:47 PM UTC

jannyfer wrote: post #92

upwardbound wrote: post #70

Earlier quoted context omitted.

Exactly right. They added too much latency because they forgot to account for the latency they added in the client. You can see from their architecture diagram (Figure 4) that the latency measurement didn't include the client delay. https://images.contentstack.io/v3/assets/bltad9188aa9a70543a... The blog post states: "The existing network monitoring system measured the latency at the networking layer as shown by the…

That kind of error would apply all the time. But the blog post states: "a calculation error that only manifested in scenarios where the actual ping was significantly lower than the target latency".

Sadly, my guess is that they are transparently lying about that. Since they apply the lag half on the client and half on the server, their lag compensation would have been off by a factor of 2x (or a factor of 100% from a relative perspective) and so they might be just claiming that a 100% error isn't very large when the total lag difference is small. "Yes we were supposed to add 2ms and we added 4ms instead, but at the end we were still only wrong by 2ms (in the other direction) which is not a big deal."

Re: Riot Games: Artificial Latency for Remote Competitors

#96

Post by bcrosby95 » Tue, May 17, 2022, 9:47 PM UTC

chabons wrote: post #85

someweirdperson wrote: post #51

Artificial or natural delays, I am somewhat confused that one end measuring/calculating ping can end up with a wrong value. Transmit at local time x, receive at local time y, subtract. It only needs a working local clock. Second surprising thing is artifical latency added on the client. Anything on the client I would avoid, if possible. Third, this all looks like magically connecting two ends. But it is packets flowi…

I think the symmetric delay here makes sense, since you're trying to simulate the latency of a (likely) symmetric connection between another client and the server. If you only add latency on the transmission, then 2 actions taken at the same time by players with different amounts of physical latency will be processed at different times. If we assume the game clocks on each client and the server are in sync, then this…

You can accomplish this by putting it all on the client, all on the server, or a mix of both (what they chose).

For all on the server, you could add latency right after receiving data from the client, but before game code processes it, and before sending data to the client.

That said, there likely were technical considerations involved in their decision to do a mix of both.

Re: Riot Games: Artificial Latency for Remote Competitors

#97

Post by chabons » Tue, May 17, 2022, 9:49 PM UTC

upwardbound wrote: post #80

chabons wrote: post #77

Earlier quoted context omitted.

I read that section as comparing what they logged before vs what the player experiences: the time taken for an input to be registered and it's effect communicated to the user. Maybe I'm wrong, but I don't think the intention was a precise description of the error, but rather the hazard of not having logs which reflect what the user experiences.

They stated that they use the same calculation for the logs as for other purposes such as latency compensation. From near the top of the blog post: "The reason we did not find it sooner is that the cause of the issue was a code bug that miscalculated latency, which meant that the values in our logs were also wrong." And from later in the post: "Our logs were not displaying the issue because the calculation was wrong.…

Right, but my point was that I think you're reading into the diagram beyond it's intended purpose. I'm not sure how you can tell that they're logging only the server-side delay, and not the client-side delay, from a systems diagram which shows each of those as a box connected by a bi-directional arrow. For instance, it's clear to me that they don't include the latency included by the game engine in that metric, but not whether they include the server-added latency or if they're aware of it and subtract it off.

Re: Riot Games: Artificial Latency for Remote Competitors

#98

Post by joebob42 » Tue, May 17, 2022, 9:49 PM UTC

jannyfer wrote: post #92

upwardbound wrote: post #70

Earlier quoted context omitted.

Exactly right. They added too much latency because they forgot to account for the latency they added in the client. You can see from their architecture diagram (Figure 4) that the latency measurement didn't include the client delay. https://images.contentstack.io/v3/assets/bltad9188aa9a70543a... The blog post states: "The existing network monitoring system measured the latency at the networking layer as shown by the…

That kind of error would apply all the time. But the blog post states: "a calculation error that only manifested in scenarios where the actual ping was significantly lower than the target latency".

I think in practice this issue doubles the added latency, which would only be noticable if the actual ping is "significantly" lower.

Because realized ping is over by as much as original ping was under, so if original wasn't significantly under...

Re: Riot Games: Artificial Latency for Remote Competitors

#99

Post by joebob42 » Tue, May 17, 2022, 9:50 PM UTC

joshuak wrote: post #68

Distributed consensus is hard. Very hard. Anyone dismissing the problem as “should never have hopped,” hasn’t a clue as to how hard and error prone reliable distributed consensus is. 35 ms lag in overall responsiveness is an eternity in competitive game play. There are videos floating around that show drawing on a tablet surface with various input latencies (perhaps someone has a link, I can’t find them at the moment…

There's no consensus involved in the bug.

Re: Riot Games: Artificial Latency for Remote Competitors

#100

Post by DantesKite » Tue, May 17, 2022, 9:52 PM UTC

colinmhayes wrote: post #73

DantesKite wrote: post #48

Earlier quoted context omitted.

The technical team acted honorably, but the upper stewardship at Riot games behaved horrendously. They sacrificed the competitive integrity of their sport because they didn't want to exclude RNG from the tournament, because they wanted the audience of the Chinese market. This article goes into more detail, giving a larger context for why this situation shouldn't have never happened in the first place. https://www.inv…

> because they wanted the audience of the Chinese market. Or maybe because the tournament would've been shit without them? Who wants to watch an international tournament with only 1 of the clearly top 2 regions represented? WHole thing would've been a forgone conclusion at that point.

Yes, the tournament would have been worse without them. What I'm saying is that does not somehow justify penalizing the other teams and the tournament writ large to "even" the playing field if you value competitive integrity.

As the article states, increasing the ping doesn't make it more fair for the other teams because they actually traveled to the event. They're physically there. It only makes it more fair for RNG who can't attend because of Covid restrictions.

To paraphrase, it "punishes the teams who are there for a team that isn't."

From a business perspective of course, I agree with you. It's definitely better to sacrifice the level of play for viewership (and it does sacrifice the level of play because artificially increasing the ping at a LAN event makes the game worse).

But if you're running a competitive eSport and you want a fair sport, it's a terrible decision.