21–30 of 82 posts
Earlier quoted context omitted.
Riot games tech blog is an awesome read. The fps (as in frames per second) performance monitoring on league of legends post recently was a great example of why its worth a read.
I thought by FPS, the OP meant First Person Shooter
Not related, but relatable. I often wonder between Games and Enterprise Software, who has a more complex backend architecture? At the outset, it seems like Games are complex as hell, in the sense that when a gamer shoots another gamer dead, there has to be freeing and re-allocation of processes/ resources, and with tens or hundreds of persons playing the same game over the internet , both the gaming architecture and…
Can you imagine anyone at a game company wanting to replatform the hit-detection system onto hadoop?
Earlier quoted context omitted.
Real reasons for: * One platform to develop on; back-end coders tend to go back and forth between client and server programming. * Faster iterations. (just hit F5 in visual studio) * IOCP Stupid reasons for: * Old IT Director denied the use of virtualisation software. (mac address randomisation wasn't great and it caused a switch crash if two people had the same mac) * Windows licenses are really cheap compared to de…
>> (until we went to cloud, where Microsoft charges insane amounts for licensing) Did you move to Azure?
But they really gouge the other cloud providers (GCP/AWS) on licensing, more than a third of our infra costs are just windows licenses (when looking only at cloud).
Our backend consists of a few somewhat large services that are broken up mostly around how they are sharded.
The biggest one is the account authority which contains most of the accout/character/item data and handles the vast majority of traffic.
We have 5 shards of that (sharding on account id) with 2 read only replicas of each one of those. All the read only requests go to one replica, and the other replica is for redundancy.
There are also other services like the party manager, ladder authority, instance manager, etc.
All of those shard on different things which is why they are seperate services.
The instance managers handle creating game instances which are the servers that the players actually play on.
We have a pile of servers which we call instance servers each of which runs an instance spawner. When it starts up the instance spawner adds its capacity to an instance manager and creates a process called a Prespawner. This prespawner loads the entire data set of the game and then waits for the instance spawner for orders.
When the instance spawner wants to create a new game instance, the prespawner runs fork() and then the new process generates it’s random game terrain which takes a few hundred milliseconds.
Because all the game resources are loaded before the fork they are all already available in memory that is shared between all of the instances running on the machine. Therefore each instance only takes 5-20 Mb of memory each which is mostly the generated terrain and monsters.
We typically run about 500 instances on the min-spec cheap Single processor Xeon servers we rent. This used to be around 1600 instances in the early days but the game got more and more CPU intensive over time as the game got more hectic over the years.
All the instances connect to Routers. There is one per instance machine, all of which connects to a few routers per data center, all of which connects to a set of core routers which also have all the backend services connected to them.
These routers are important because they know where everything and everyone currently is.
The routers work sort of like internet routers work, but instead of IP addresses, you address your requests to logical entities or groups which can move around and the router network is tasked with keeping track of.
So for example, when you whisper someone, you are sending a message to Account:123 and it will find it’s way to whatever server currently has Account:123 on it right now. If you send a message in global chat to GlobalChat:1 it will be multicasted through the network to all the servers which have currently registered an interest in hearing GlobalChat:1.
If you add someone to your friend list, then what that means is the server you are on will register interest in multicast group AccountSession:123 which is a group that account 123 will multicast all its status updates to like moving between zones or leveling up or whatever.
Parties, Leagues, Guilds, Etc, etc. All of these things have multicast groups associated with them.
If you have any more questions then feel free to ask.
var player = new Player(123);
await player.FireWeapon();
In this way, it is very easy to quickly update game state without obsessing over the throughput of a single machine.
Halo 4 and 5 use a project developed “in house” at Microsoft called Orleans. Roughly speaking it is similar to Akka. It’s an actor based distributed system which attempts to hide the implementation of distributed networking. In essence, each match and each player get their own “network attached“ object (a “grain” in Orleans terms.) So: var player = new Player(123); is actually instantiated _somewhere_ on the network…
It took me quite some time to shift my mental model and decide I prefer it over Akka. It's also worth noting that Orleans is open source. There is also an excellent operational dashboard named OrleansDashboard.
This question sounds like it's pointed directly at me. However, I can only speak for one AAA gaming company, and my team operate a bit differently than most in the company. My team operates the infrastructure for "Tom Clancy's The Division" video game series (1&2). Most of the programming approach is spent on doing the cheapest (in terms of CPU) possible thing, everything is C++ Things like: matchmaking will happen i…
Among the unique features are globally distributed hot-deploy of server side code/resources and globally distributed JSON database over HTTP. It's completely async. concurrent and all threads work on all memory which is not what most systems have if you look under the hood.
Only Java (begin the down vote without arguments) has good enough non-blocking IO and memory model for networked concurrency to be workable on the server side, but most implementations are bloated.
The real discussion we should have is tick vs event based protocols, I'm 100% convinced we need to move away from tick based protocols.
Earlier quoted context omitted.
What is the rationale behind using Windows on the servers? I'm wondering because the argument I usually hear is that someone high up in the ops team hierarchy declares "but we need everything in the AD, including all the servers, otherwise I can't sleep well". In your case however this apparently doesn't apply, as you do not register them in the AD. Is it so the game devs don't need to write cross-platform server cod…
Real reasons for: * One platform to develop on; back-end coders tend to go back and forth between client and server programming. * Faster iterations. (just hit F5 in visual studio) * IOCP Stupid reasons for: * Old IT Director denied the use of virtualisation software. (mac address randomisation wasn't great and it caused a switch crash if two people had the same mac) * Windows licenses are really cheap compared to de…