Live data from Hacker News

The unlikely story of Teardown Multiplayer

blog.voxagon.se

31–40 of 69 posts

Re: The unlikely story of Teardown Multiplayer

#31
post #19

Earlier quoted context omitted.

That's true, I'm not saying it comes without trade-offs. But in return you get a perfectly consistent and physically accurate simulation. It would mostly be expensive, I think, but it's technically feasible (services like Shadow or GeForce Now already demonstrate that).

Most consumer GPUs have a limit on the number of video streams their hardware encoder can handle at once, and in some cases the limit is as low as 2.

Okay, I didn't know that

Re: The unlikely story of Teardown Multiplayer

#32
post #23

I really love Teardown, and I’m absolutely baffled at the bizarre performance that game manages to push on even very large worlds. I’ve tried to load teardown levels in a homegrown engine and I always end up stuttering like hell as soon as GI becomes involved (or even before that). I’m going to finally manage to replicate it, and then the new engine will be released and raise the bar again xD

Did you check out the "about" section and the timeline of his career on the main page of the linked blog?

https://www.voxagon.se/

Because it looks like your opponent is a Swedish former demoscener who started programming at age 12 on the C64 and Amiga computers in 1990, quickly moving on to writing games and demos in assembly, then professionally developing physics engines since 2001, specializing in game performance profiling and squeezing performance out of optimized mobile games.

As far as game dev stereotypes go you basically picked a Final Boss fight. Good luck, you'll need it :p

Re: The unlikely story of Teardown Multiplayer

#33

I feel like there's detail missing in the blog. The "Reliable vs Unreliable" section implies that different parts of the scene are sent using a strict-ordering protocol so that the transforms happen in the same order on every client, but other parts happen in a state update stream with per client queueing. But which is which? Which events are sent TCP and which are UDP (and is that literally what they're doing, or on…

They didn't explain those in detail because they didn't implement these concepts themselves. The GameNetworkingSockets library (and most likely also the internal network backend that was mentioned) provides them: > A reliability layer significantly more sophisticated than a basic TCP-style sliding window. It is based on the "ack vector" model from DCCP (RFC 4340, section 11.4) and Google QUIC and discussed in the context of games by Glenn Fiedler. The basic idea is for the receiver to efficiently communicate to the sender the status of every packet number (whether or not a packet was received with that number). By remembering which segments were sent in each packet, the sender can deduce which segments need to be retransmitted. https://github.com/ValveSoftware/GameNetworkingSockets

Re: The unlikely story of Teardown Multiplayer

#34
post #4

"1.Serialize the entire scene, compress the data, and pass it to the joining client. We already do full scene serialization for quicksave and quickload, so this is possible, but the files are large: 30-50 MB is common, often more, so transfer would take a while. [...] 3. Record the deterministic command stream, pass it to the joining client, and have that client apply all changes to the loaded scene before joining th…

Part of the problem with [1] is that you still end up needing [3] anyhow, because even if you've got a fiber-to-fiber connection, while the transfer was occurring the game world has moved on and you'll need to replay that anyhow.

But if you've got a solution for [3] that works completely correctly anyhow, then writing lots of code for [1] becomes redundant to that anyhow, even with save/load code sitting right there. Might as well start from the beginning and replay it anyhow.

One of the things I will often do that I sometimes have to explain to my fellow engineers is bound the rate at which we'll have to handle certain changes based on what is making them. If you know that you've got a human on the other end of some system, they can only click and type and enter text so quickly. Yes, you still need to account for things like copy & paste if that's an issue for your system, where they may suddenly roll up with a novel's worth of text, but you know they can't be sending you War and Peace sixty times a second. You can make a lot of good scaling decisions around how many resources a user needs when you remember that. The bitrate coming out of a human being is generally fairly small; we do our human magic with the concatenation of lots of bits over lots of time but the bitrate itself is generally small. For all that Teardown is amazingly more technically complicated than Doom, the "list of instructions the humans gave to the game" is not necessarily all that much larger than a Doom demo (which is itself a recording of inputs that gets played back, not a video file), because even Doom was already brushing up on the limits of how many bits-per-second we humans can emit.

Re: The unlikely story of Teardown Multiplayer

#35

I feel like there's detail missing in the blog. The "Reliable vs Unreliable" section implies that different parts of the scene are sent using a strict-ordering protocol so that the transforms happen in the same order on every client, but other parts happen in a state update stream with per client queueing. But which is which? Which events are sent TCP and which are UDP (and is that literally what they're doing, or on…

They didn't explain those in detail because they didn't implement these concepts themselves. The GameNetworkingSockets library (and most likely also the internal network backend that was mentioned) provides them: > A reliability layer significantly more sophisticated than a basic TCP-style sliding window. It is based on the "ack vector" model from DCCP (RFC 4340, section 11.4) and Google QUIC and discussed in the con…

How did you find out their game used this library? Maybe I missed it but didn’t seem to be mentioned anywhere

Re: The unlikely story of Teardown Multiplayer

#36
post #14

> For the longest time (and for good reasons), floating point operations were considered unsafe for deterministic purposes. That is still true to some extent, but the picture is more nuanced than that. I have since learned a lot about floating point determinism, and these days I know it is mostly safe if you know how to navigate around the pitfalls. If you're only concerned about identical binaries on x86, it's not t…

As far as I know, the ARM (at least aarch64) situation should be about the same as x86-64. Anything specific that's bad about it? (there's aarch32 NEON with no subnormal support or whatever, but you can just not use it if determinism is the goal)

that RECIP14 link is AVX-512, i.e. not available on a bunch of hardware (incl. the newest Intel client CPUs), so you wouldn't ever use it in a deterministic-simulation multiplayer game anyway, even if you restrict yourself to x86-64-only; so you're still stuck to the basic IEEE-754 ops even on x86-64.

x86-64 is worse than aarch64 is a very important aspect - baseline x86-64 doesn't have fused multiply-add, whereas aarch64 does (granted, the x86-64 FMA extension came out around not far from aarch64/armv8, but it's still a concern, such is life). Of course you can choose to not use fma, but that's throwing perf away. (regardless you'll want -ffp-contract=off or equivalent to make sure compiler optimizations don't screw things up, so any such will need to be manual fma calls anyway)

Re: The unlikely story of Teardown Multiplayer

#37
post #21
post #13

Earlier quoted context omitted.

Which one of your friends can host an mp physics heavy game with a number of low-latency high-resolution video streams? I would estimate the average answer to be zero.

Perhaps the solution could be to have all players stream the game from a centralized instance, rather than all clients streaming from the host’s instance. That would have a number of advantages, come to think of it. For starters, install size could be much lower, piracy would be a non-issue, and there would be no need to worry about cross-platform development concerns.

We don't have to theorize about this. We've had cloud gaming for years, and the companies have immense motivations to turn us all into renters in the cloud so they've poured a lot of effort into it and we can see half-a-dozen highly-resourced results now. We can just look at it and we can see that it... almost... works. If you don't care much about latency it definitely works.

However, Teardown is in the set of games where it just barely works and only if all the stars and the moon align. I'd characterize it as something like, cloud gaming spends 100% of the margin, so if anything, anything goes wrong, it doesn't work very well.

(Plus, as excited as the companies are about locking us into subscriptions rather than purchases that we own, when it comes time to actually pay for the service they are delivering they sure do like to skimp, because it turns out it's kind of expensive to dedicate the equivalent of a high-end gaming console per person. Most stuff that lives in the cloud, a single user averages using a vanishing fraction of a machine over time, not consuming entire servers at a time. Which doesn't pair well with "you spent 100% of the margin just getting cloud gaming to work at all".)

Re: The unlikely story of Teardown Multiplayer

#38

Wouldn't it have been simpler (even if technically heavy) to host the game on a single machine and just stream each player's camera? That way all the physics would be computed in real time on one computer, and each player would just receive a different video stream.

This seems to get brought up in every hn post on video game mulriplayer, and it makes me wonder: do you play video games? I dont know of any video games that do multiplayer that way, and i would think that alone suggests its not a good idea. Who wants to play a game with 50ms+ keypress to screen update delay? Sounds miserable.

It makes me laugh whenever there's a post about anticheat on the frontpage and without missing a beat there's always a comment there - "why don't they just run the entire game logic server side and stream the updates to the client??? are they stupid?"

Re: The unlikely story of Teardown Multiplayer

#39

Earlier quoted context omitted.

This seems to get brought up in every hn post on video game mulriplayer, and it makes me wonder: do you play video games? I dont know of any video games that do multiplayer that way, and i would think that alone suggests its not a good idea. Who wants to play a game with 50ms+ keypress to screen update delay? Sounds miserable.

I don't play it much anymore, but I used to be a heavy player: the latency isn't 50 ms on GeForce Now (with a French connection, which is pretty good).

yea thats barely playable for single player games

Re: The unlikely story of Teardown Multiplayer

#40
post #14

> For the longest time (and for good reasons), floating point operations were considered unsafe for deterministic purposes. That is still true to some extent, but the picture is more nuanced than that. I have since learned a lot about floating point determinism, and these days I know it is mostly safe if you know how to navigate around the pitfalls. If you're only concerned about identical binaries on x86, it's not t…

I'm pretty sure he is talking about deterministic output.
Post reply on HN