Live data from Hacker News

MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

prdeving.wordpress.com

21–30 of 76 posts

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#21
> the I/O bottleneck in the database

I thought most MMOs kept everything in memory and only offloaded to the database periodically. I clearly remember rollbacks to fix times (XX:00) when things went down.

Edit: Sorry, should have read the whole article before commenting.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#22

Does anyone know any more good resources for designing an MMO architecture? Would love to do a MMO as a side project but a bit daunted by the unknown of architecture development.

I'd wager a guess that getting players will be the most difficult part by far, at least in the beginning. Make an MVP and focus on building a playerbase first, then come back to architecture when you're suffering from success if you get that far.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#23

Does anyone know any more good resources for designing an MMO architecture? Would love to do a MMO as a side project but a bit daunted by the unknown of architecture development.

I've some other posts planned about this topic, I don't know when or even if im going to deliver, but you are free to follow the blog and receive the update if I ever do.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#25
post #9

Earlier quoted context omitted.

The WoW development diary talks abit about their server code and how they handle the load.

Do you have a link? I’m intrigued.

Its a book https://whenitsready.com/wowdiary/ originally sold as a kickstarter by one of the original Blizzard mappers made from notes they wrote while working on original wow. They did some AMAs[1][2] with other wow developers that sort of touch on some of the topics discussed here.

[1] https://www.reddit.com/r/wow/comments/9huows/ama_former_wow_...

[2] https://www.reddit.com/r/classicwow/comments/9fb2bo/john_sta...

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#26
post #22

Does anyone know any more good resources for designing an MMO architecture? Would love to do a MMO as a side project but a bit daunted by the unknown of architecture development.

I'd wager a guess that getting players will be the most difficult part by far, at least in the beginning. Make an MVP and focus on building a playerbase first, then come back to architecture when you're suffering from success if you get that far.

> then come back to architecture when you're suffering from success if you get that far.

Then it will be too late because you will essentially have to rewrite half of your project while your userbase is leaving due to unplayable game. Better to make good architectural decisions from the start, and make small optimizations when needed.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#27

Does anyone know any more good resources for designing an MMO architecture? Would love to do a MMO as a side project but a bit daunted by the unknown of architecture development.

I made a video series on networking theory for virtual worlds that has been well received: https://youtu.be/0wOZusuMIIM

I've also been working on an engine for the past few years if you want some code examples: https://github.com/Net5F/AmalgamEngine

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#28
post #21

> the I/O bottleneck in the database I thought most MMOs kept everything in memory and only offloaded to the database periodically. I clearly remember rollbacks to fix times (XX:00) when things went down. Edit: Sorry, should have read the whole article before commenting.

Exactly, there is no way the state can be persisted on absolutely every change. It has to be periodically dumped to the database.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#29
The "cache, lots of cache" statement is the most true of any MMO architecture we can build. I did some optimization work earlier this year on a project where the single back-end server is now handling 2 billion requests per minute and had around 3TB of RAM for cache (I think the final production system was aiming for 12TB of RAM).

There's concerns around race conditions as you pointed out, message passing from client to server, and server to server, client hand-off between sharded servers. Those synchronization problems will haunt your dreams.

I think the biggest issue I still struggle with is tracking those ephemeral problems that only happen on one shard, or only when going between this shard and this shard, but not the other way. One useful trick is obviously message prioritization and different messages heading to different servers - though these days I'd put a message router in front of the shards and the router handles persistent connections other than the usual technique of direct connection I've employed in the past.

Contributed to an MMO game that involves waving light sabers around, another where you defeat the ultimate prime evil (though I was more on the fraud detection on that one), an unpublished MMO that unceremoniously died during the 2008 financial crash, a "shared world" game that involved animals, an open-world game that involves driving cars and running pedestrians over, a few "internet scale" websites, and am currently lead back-end on another MMO - though our database requirements are relatively simple this time around, but it is still again, read-at-start-up, write-only-when-necessary.

Re: MMO Architecture: Source of truth, Dataflows, I/O bottlenecks and how to solve

#30
I wrote about some of these issues client-side, in a previous post about a Rust metaverse client. This is a much worse problem in a metaverse system, because there are no static game level maps. Every object in the world is in a database somewhere.

Second Life / Open Simulator makes a big distinction between assets, inventories and area state. Assets (meshes, textures, animations, sounds) are immutable, and are stored more or less permanently. (There's a garbage collection batch job that runs monthly or so) Those are basically files. There's a vanilla web service running on an AWS web server, and Akamai, both heavily cached.

Inventories are like file directories. They have asset UUIDs and some metadata (name, etc.) Those are in a database, but that data is dynamic and not cached. Each user has an inventory, of course, and it can be huge. 50,000 items are not unheard of. This is a metaverse; you can build stuff.

Area state is in server memory for each region. That's saved periodically, once a minute or so. This is a backup file, not a database. If you wanted a more continuous save process, you could keep a log of recent changes on a different machine than the server. After a crash, reload the server state and rerun the recent changes. Area state is under a gigabyte per region (A region is 256x256 meters).

With a three level system like this, none of the levels are severely overloaded. The greatest data volume is from the asset store, and because that's immutable, it can be and is cached extensively. There are three levels of caches - asset server, CDN, and client. It's still a problem getting assets out to the clients fast enough, but with prioritization and concurrency, that's solveable. The inventory database is mostly-read, so the usual scaling techniques for mostly-read databases work. Area state is in memory. The main trick is taking a clean backup without visibly freezing the system.

Post reply on HN