Live data from Hacker News

MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

prdeving.wordpress.com

31–40 of 76 posts

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#31
In most data architectures the DB is only the backing store, because no matter how fast your database is it's going to be slower than RAM.

Once you start caring about the performance the second thing you do is stick a cache layer of one sort or another in front of the database; the first thing should be making sure you have the correct indexes.

In any case, it sounds like a distributed cache problem. I wonder if you could just abuse redis for your game backend?

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#32
post #31

In most data architectures the DB is only the backing store, because no matter how fast your database is it's going to be slower than RAM. Once you start caring about the performance the second thing you do is stick a cache layer of one sort or another in front of the database; the first thing should be making sure you have the correct indexes. In any case, it sounds like a distributed cache problem. I wonder if you…

may help to read the article, redis is mentioned

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#33
post #10

Earlier quoted context omitted.

to be honest, i know nothing about private WoW servers but i promise i'll check it out! Thanks!

azerothcore is probably the best and most polished, if you don't mind wotlk

the last expansion that was any good?

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#34
Speaking about WoW specifically, since I'm not familiar with the others, I've always been curious about their quest system. Specifically keeping track of what are available for the character efficiently along with the event system to flag quests as completed, etc.

There's so many of them. I have to assume they're spatially limited. You enter a zone, or an area, and the system loads up all of the quests located in that space, then it runs through to determine whether you qualify for them.

As for quests that you're on, that's a bit more straightforward, since you're so limited to how many you can carry around with you at any one time. Then, every event can practically just be brute forced across your pending quests to see which ones get progressed, etc.

But it was always a curiosity to me considering the magnitude of the quests available how most anything can trigger quest progress.

There's also the whole achievement system, which perhaps is similar in design.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#35
post #5

Early MMOs (WoW, Asheron's Call, Everquest, Daoc) were very impressive in terms of distributed computing.

WoW is not in the same league as EQ, AC, DAOC.

WoW is a 2nd or if you count Meridian 59 as gen1, 3rd generation MMORPG.

It was not the 1st that had seemless maps, but IMHO it did that best. EQ2, while not having seamless maps, would be in the same league as WoW.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#36
post #31

In most data architectures the DB is only the backing store, because no matter how fast your database is it's going to be slower than RAM. Once you start caring about the performance the second thing you do is stick a cache layer of one sort or another in front of the database; the first thing should be making sure you have the correct indexes. In any case, it sounds like a distributed cache problem. I wonder if you…

You can use redis or memcached, but every MMO or online game I've been involved with, unless it was a "web game", has eschewed those for the most part. The game server maintains the state, knows all the objects in the universe, or at least its portion of the universe, and is responsible for retrieving and updating those objects. Even redis and memcached would be considered slow by comparison. Those game objects/world objects/MOBs may eventually be pushed out to a key-store server, but generally are not. The only portion of the database on any MMO I've worked on that has cared about "proper indexes" has been the area dealing with account retrieval. Traditional databases, at least on the non-web MMOs I've been involved with, when it comes to game state, are not normally used. RDBMS are used for boring things like account management, customer management, and so forth. Our database on the current (non-web) MMO uses a few more web technologies than I have in the past for this particular problem, but once the shard is loaded, and the user is connected, it is back to tradition, for the most part.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#37
post #20
post #6

Earlier quoted context omitted.

They were not really distributed though, I think one of the first that really started was Guild Wars 2. https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2017/Pres... DAoC was basically a Linux box with a bunch of processes connected to MySQL. https://www.gamedeveloper.com/disciplines/postmortem-mythic-...

Would this count? https://www.gamedeveloper.com/design/classic-postmortem-i-as... > One the most impressive features of the Turbine engine is the continuous outdoor environment. This is made possible thanks to dynamic load balancing, which is a scalable serverside architecture. The easiest way to appreciate the need for dynamic load balancing is to consider the following scenario. > Dynamic load balancing solves this…

That's a useful technology. Second Life / Open Simulator do not have that, and need it. It's good to hear about a success with that approach.

Improbable tried that, dividing the world into regions but moving the region boundaries around based on player density. This worked, but apparently required huge amounts of inter-server traffic. The system was too expensive to operate. (Running it on Google Cloud with metering for every client/server transaction didn't help.) Five indy free to play games, some of them good (look up Worlds Adrift), went bust because of server cost.

Improbable then pivoted to simulators for the UK military, a much less cost-sensitive market. That worked, but they had way too much company and funding for that niche. Then they tried to pivot to crypto metaverses, two years too late, and hooked up with the Yuga Labs (Bored Ape, Otherside) crowd. Lately, they're trying to do something with US Major League Baseball. Their solution to the cost problem is to only run special events that last a few hours, for which they can short term rent some huge number of servers from AWS or somebody.

There's still no good off the shelf solution for this kind of scaling, with big worlds and big moving crowds. Epic and Roblox were making noises about working on this problem a year ago, but not much has been heard recently. Now both are in money-losing and layoff mode.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#38

Does anyone know any more good resources for designing an MMO architecture? Would love to do a MMO as a side project but a bit daunted by the unknown of architecture development.

The basic advice in the article is sound really, make writes to db very async and keep all your state in memory.

Also important for most online games: any core gameplay relevant actions need to be server authenticated - usually you do RPCs from client to server, resolve the result, then broadcast it to the relevant clients.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#39

The "cache, lots of cache" statement is the most true of any MMO architecture we can build. I did some optimization work earlier this year on a project where the single back-end server is now handling 2 billion requests per minute and had around 3TB of RAM for cache (I think the final production system was aiming for 12TB of RAM). There's concerns around race conditions as you pointed out, message passing from client…

How did they get the architecture so wrong on that "open-world game that involves driving cars and running pedestrians over"?

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#40
post #6
post #5

Early MMOs (WoW, Asheron's Call, Everquest, Daoc) were very impressive in terms of distributed computing.

They were not really distributed though, I think one of the first that really started was Guild Wars 2. https://ubm-twvideo01.s3.amazonaws.com/o1/vault/gdc2017/Pres... DAoC was basically a Linux box with a bunch of processes connected to MySQL. https://www.gamedeveloper.com/disciplines/postmortem-mythic-...

To be fair, GW2 was able to do that by instancing the world per zone, with loading screens to switch server connections, rather than having a seamless open world.
Post reply on HN