I’d be really interested in reading about multiplayer servers for big title FPS games with matchmaking and lots of players
Ask HN: What is the ops architecture like for AAA multiplayer game servers?
1–10 of 82 posts
Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?
#2Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?
#3Not FPS specifically, but you might enjoy Riot Games' tech blog https://technology.riotgames.com/ which has articles on a variety of game technology things such as service deployment, network infrastructure, game performance monitoring etc.
(Disclosure: I work there)
Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?
#4Not FPS specifically, but you might enjoy Riot Games' tech blog https://technology.riotgames.com/ which has articles on a variety of game technology things such as service deployment, network infrastructure, game performance monitoring etc. (Disclosure: I work there)
Riot games tech blog is an awesome read. The fps (as in frames per second) performance monitoring on league of legends post recently was a great example of why its worth a read.
Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?
#5Not FPS specifically, but you might enjoy Riot Games' tech blog https://technology.riotgames.com/ which has articles on a variety of game technology things such as service deployment, network infrastructure, game performance monitoring etc. (Disclosure: I work there)
Hello fellow Rioter!
Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?
#6This question sounds like it's pointed directly at me.
However, I can only speak for one AAA gaming company, and my team operate a bit differently than most in the company.
My team operates the infrastructure for "Tom Clancy's The Division" video game series (1&2).
Most of the programming approach is spent on doing the cheapest (in terms of CPU) possible thing, everything is C++
Things like: matchmaking will happen ideally on a single machine with no shared state, everything will happen in memory, which is much faster and can be more reliable than any distributed state or failover.
(it's less reliable if you're in matchmaking and the server or service dies; But then everyone's client will reconnect to the newly scheduled matchmaking instance and populate in memory state.)
We use a lot of windows, nearly every machine that doesn't handle state is a windows server. This has pros and cons, from my ops perspective I try to treat windows like cattle, but windows doesn't like that. they have their own way of operating fleets of machines which include using SCCM and packaging things. There's nice GUI's, but we use saltstack and we removed AD, because it was a huge burden to: create a machine, link it to AD, reboot it, finally get a machine worth using.
From a dev perspective, Windows is good, IO Completion Ports is superior to the linux epoll in terms of interface and performance, so we can have machines that take 200,000+ connections.
How you decide which dedicated server you connect to is up to your client, it does a naive check as it's logging in where it will do a TLS handshake with a random gameserver in a region, for each region. (during the login phase we send your client a list of all currently active gameservers and an int to represent the region).
This works fine until there's packet loss on a particular path because your single ping might be fine but overall your experience could be better elsewhere; if you're not able to ping anything then we fall back on geoip.
That said, if you have friends on another datacenter, we try to put you on the same server. So that if you join groups or whatever then it's just a phase transition rather than a full server sync.
Everything is orchestrated with a "core" set of servers which handle player authentication and matchmaking centrally, then each of the gameserver datacenters (geographically distributed to be closer to players) is attached via ipsec VPN.
In Division 2 we spread out into GCP as well as having physical machines, so we developed a custom auto-scaler. The autoscaler checks the capacity of a region and how many players are currently in a region, keeps a record over 10 minutes and makes a prediction. If the prediction goes over the current capacity in 20 minutes or less, it will create a new instance (since making windows servers on GCP takes longer than linux servers)
If the prediction goes lower than the capacity of a server, it will send a decomission request to the machine, which takes up to 3hrs to complete (to give people time to leave the server naturally).
Idk, I've been doing this for 5 years now so I can talk at length about how we do it, but ultimately out biggest challenges are the fact that we can't use cool shit or new trends because latency matters a lot and we use windows everywhere.
--
As an aside; the overwhelming majority of other ubisoft games (excption: For Honor) use something very similar to what we released open source in collaboration with google to do matchmaking: https://agones.dev/site/
Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?
#7Some network related info from the pubg devs
https://steamcommunity.com/games/578080/announcements/detail...
Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?
#8There's a slidedeck and presentation that's been posted to HN a number of times about Call of Duty's servers using Erlang.
https://www.erlang-factory.com/upload/presentations/395/Erla...
https://news.ycombinator.com/item?id=14120506
Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?
#9This question sounds like it's pointed directly at me. However, I can only speak for one AAA gaming company, and my team operate a bit differently than most in the company. My team operates the infrastructure for "Tom Clancy's The Division" video game series (1&2). Most of the programming approach is spent on doing the cheapest (in terms of CPU) possible thing, everything is C++ Things like: matchmaking will happen i…
What is the rationale behind using Windows on the servers? I'm wondering because the argument I usually hear is that someone high up in the ops team hierarchy declares "but we need everything in the AD, including all the servers, otherwise I can't sleep well". In your case however this apparently doesn't apply, as you do not register them in the AD.
Is it so the game devs don't need to write cross-platform server code?
Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?
#10This question sounds like it's pointed directly at me. However, I can only speak for one AAA gaming company, and my team operate a bit differently than most in the company. My team operates the infrastructure for "Tom Clancy's The Division" video game series (1&2). Most of the programming approach is spent on doing the cheapest (in terms of CPU) possible thing, everything is C++ Things like: matchmaking will happen i…
What is the rationale behind using Windows on the servers? I'm wondering because the argument I usually hear is that someone high up in the ops team hierarchy declares "but we need everything in the AD, including all the servers, otherwise I can't sleep well". In your case however this apparently doesn't apply, as you do not register them in the AD. Is it so the game devs don't need to write cross-platform server cod…
Real reasons for:
* One platform to develop on; back-end coders tend to go back and forth between client and server programming.
* Faster iterations. (just hit F5 in visual studio)
* IOCP
Stupid reasons for:
* Old IT Director denied the use of virtualisation software. (mac address randomisation wasn't great and it caused a switch crash if two people had the same mac)
* Windows licenses are really cheap compared to developer time. (until we went to cloud, where Microsoft charges insane amounts for licensing)