Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What is the ops architecture like for AAA multiplayer game servers?

news.ycombinator.com

Post reply on HN
Print view View original

71–80 of 82 posts

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#71

Post by druerridge »

Spent half a dozen years working in AAA realm for multiplayer games and an equal amount of time working on my own indie projects, and as you can see already in this thread, there's really two (often confused) pieces to this conversation. First, there's Multiplayer Gameplay Engineering. This is typically a single process handling the 2~64 people shooting each other in a single game. Second, there's Online Services Engineering for Games. This is typically orchestrating the above process, hundreds or thousands of times, as well as things like Matchmaking systems, party systems, Storefronts, etc.

Below are two articles which I think can be really valuable "Baby's first" for the topics of Multiplayer Gameplay Engineering and Online Services for Games. I wrote them beginning from my perspective of approaching these problems my first time as a college student many years ago, and continued with how my approach evolved through experience. I also include lots of links to seminal articles/talks on the topics that I read as I attempted to "do my homework" while making my own indie game.

Multiplayer Gameplay Engineering https://www.gamasutra.com/blogs/DruErridge/20181004/327885/B...

Online Services for Games https://www.gamebreakingstudios.com/posts/dedicated-game-ser...

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#72

Post by druerridge »

dijit wrote: post #6

This question sounds like it's pointed directly at me. However, I can only speak for one AAA gaming company, and my team operate a bit differently than most in the company. My team operates the infrastructure for "Tom Clancy's The Division" video game series (1&2). Most of the programming approach is spent on doing the cheapest (in terms of CPU) possible thing, everything is C++ Things like: matchmaking will happen i…

Speaking to language usage (just as one topic) for Online Services for Games. I now contract in the games services space (after half a dozen years at a AAA studio) and have gotten to see a bit of an overview of how folks approach it. For small and medium studios, I see a lot of Python/Django approach to the problem which is quick to develop in and has a lot of problems solved for you. I also think Python is a frequent choice because a lot of the engineers are pulled from gameplay engineering, and python is familiar to a lot of folks from scripting their tools or plugins (e.g. Maya). Node/express pokes its head out for some small studios as well for similar reasons to Django. Go is increasingly popular for both AAA and indie companies for pretty obvious reasons (online services is literally what it was built for). C++ and then Java/Scala are common choices for older and more established studios.

In my experience, all of these languages are fairly well equipped to the problem space - even the interpreted/garbage collected ones can handle these problems well as they are (in my experience) infrequently CPU bound, and frequently the scaling considerations come in around your usage of databases and caches (network or IO bound). When you have enough players to hit CPU boundaries, you also typically have enough money to buy some pretty beefy CPUs. The caveat to this is that concurrency gives huge wins, since a lot of the requests/data are not related to each other (beyond the 10-20 people in a game) and so languages or frameworks that don't solve concurrency and/or async IO well are at a disadvantage. This puts Python/Django and Node.js further down the totem for me. Of course, there are ways to resolve those problems even within those languages so they are far from ill-equipped.

When I get my choice, I choose something garbage collected in a VM, with mature web frameworks (basically Java & C#) for online services in games. Knowing an engineer's errant null pointer isn't going to tip over your whole process is pretty handy, and the CPU gains in C++ or Go (native code) aren't put to as good of use here as they are in gameplay engineering where you're trying to squeeze cycles out of client machines. Client machines don't really get more powerful the more players you have, but your servers can, haha.

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#73

Post by hermitdev »

scarface74 wrote: post #39
speedplane wrote: post #12

Earlier quoted context omitted.

I no longer do windows development, but I miss it. It's strengths are greater than you portray. The C# / .NET libraries are really beautiful and well thought out. I most often program in Python, and while the Python language is more concise and elegant than C#, the Python libraries are not nearly as well organized and consistent as C# / .NET. Java libraries are better than Python but worse than C#, and the Java langu…

I was an exclusive Microsoft developer for a little over 20 years. It wasn’t until I started doing cloud deployments - like the original poster implied - that I started avoiding Windows as often as I could. Once you add Windows to the mix, everything gets worse - startup time, resource requirements, automation, snd costs. Luckily, you can do C# without having to use Windows with .Net Core.

Curious, how've your cross platform .Net Core experiences been? I've not done much with Linux, but when I was benchmarking writing to MS SQL with the same binary on Linux & Win10, .Net Core had about half the throughput to a remote SQL server instance than Win10. Not sure if others have seen this, but .Net Core seems to run significantly slower on linux vs Windows, in my experience.

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#74

Post by hermitdev »

godelmachine wrote: post #19

Not related, but relatable. I often wonder between Games and Enterprise Software, who has a more complex backend architecture? At the outset, it seems like Games are complex as hell, in the sense that when a gamer shoots another gamer dead, there has to be freeing and re-allocation of processes/ resources, and with tens or hundreds of persons playing the same game over the internet , both the gaming architecture and…

Games are complex, but nothing compared to large enterprise systems, especially financial.

Typical (simplified) day for me: either download files or expect files to be delivered by start of business (typically 8am). Files need to be cleaned, uploaded. Changes need to be sent to untold numbers of other systems. 830am-300pm, various trading activity, new instruments being added, quants want new benchmarks, etc, more data loaded. 3pm, trading ends, mad rush to calc end of day returns, pricing and positions. All of this has to be sent to external accountants and auditors within about an hour.

Id imagine games largely dont have many external dependencies (maybe on a service such as steam, Xbox live, PS network, etc), but they dont have to process and or deliver files from/to numerous sources for tje game to work. They just need the service live. They dont have to wait 15 minutes to check Bloomberg SFTP to find out that their request failed for some untold reason.

I wouldn't be surprised to find AAA gaming and financial networking to be similarly, but differently complex for different and yet similar reasons. Both will want to reduce latency to the greatest extent possible, but for very different reasons.

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#75

Post by scarface74 »

hermitdev wrote: post #73
scarface74 wrote: post #39

Earlier quoted context omitted.

I was an exclusive Microsoft developer for a little over 20 years. It wasn’t until I started doing cloud deployments - like the original poster implied - that I started avoiding Windows as often as I could. Once you add Windows to the mix, everything gets worse - startup time, resource requirements, automation, snd costs. Luckily, you can do C# without having to use Windows with .Net Core.

Curious, how've your cross platform .Net Core experiences been? I've not done much with Linux, but when I was benchmarking writing to MS SQL with the same binary on Linux & Win10, .Net Core had about half the throughput to a remote SQL server instance than Win10. Not sure if others have seen this, but .Net Core seems to run significantly slower on linux vs Windows, in my experience.

Most benchmarks show either similar or better performance overall.

https://stackoverflow.com/questions/44334125/net-core-on-win...

https://robertoprevato.github.io/More-about-Linux-vs-Windows...

But there are reported issues with sql:

https://github.com/dotnet/corefx/issues/24480

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#76

Post by midniteslayr »

markmandel wrote: post #51

If you want to learn more about this in person, in 2020 there will be the first "Online Game Technology Summit" at the Game Developers Conference. (Disclaimer: I'm one of the summit advisors) We're trying to grow the education in this space of game development, because currently it is very sparse. AFAIK, this is the only event that is dedicated to the technical aspects of online, connected or multiplayer games. Detai…

I'm probably gonna submit a talk to this summit this year! Been going to GDC as a volunteer since 2005, I've always wanted to talk more about online services development for sometime. So glad to see this happening!

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#77

Post by ezekg »

cbartlett wrote: post #7

Some network related info from the pubg devs https://steamcommunity.com/games/578080/announcements/detail...

This is a great read. I always wondered if AAA battle royale games did something like that, since updating distant players every frame, especially if they're behind the local player, seems like a waste.

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#78

Post by ezekg »

lwansbrough wrote: post #25

Halo 4 and 5 use a project developed “in house” at Microsoft called Orleans. Roughly speaking it is similar to Akka. It’s an actor based distributed system which attempts to hide the implementation of distributed networking. In essence, each match and each player get their own “network attached“ object (a “grain” in Orleans terms.) So: var player = new Player(123); is actually instantiated _somewhere_ on the network…

Halo 5 has some of the best net code I've encountered recently. The hit detection really is top notch compared to other FPS games.

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#79

Post by Accujack »

lucifirius wrote: post #43

Here's some EVE Online links. Really amazing stuff. https://www.eveonline.com/article/tranquility-tech-3 http://highscalability.com/eve-online-architecture

Seconded. Eve's server architecture is amazing given the original constraints of the network... they managed to find a way to make a single worldwide server work and to make the game treat everyone fairly no matter what their ping was.

The architecture used has limitations (the biggest one is that fps-style real time isn't really possible) but it's aged very well.

Re: Ask HN: What is the ops architecture like for AAA multiplayer game servers?

#80

Post by [deleted] »

dijit wrote: post #23
godelmachine wrote: post #20

Earlier quoted context omitted.

>> (until we went to cloud, where Microsoft charges insane amounts for licensing) Did you move to Azure?

No, it's cheaper on Azure (and they give us some huge discounts because we're so MS heavy). But they really gouge the other cloud providers (GCP/AWS) on licensing, more than a third of our infra costs are just windows licenses (when looking only at cloud).

[deleted]
Post reply on HN