Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How we built an auto-scalable Minecraft server for 1000+ players

worldql.com

Post reply on HN
Print view View original

61–70 of 145 posts

Re: How we built an auto-scalable Minecraft server for 1000+ players

#61

Post by lwansbrough »

Hmm, I still think spatially allocated servers offer the best scalability. You would just have to approach it a little differently.

Server boundaries could move based on where the population is via delaunay triangulation (instead of fixed boundaries), and servers could share high-importance information with their immediate bordering neighbours. (This would be recognized as ghost data on the neighbouring servers.) You could even go further and have neighbours share ghost data with their other neighbours at a lower fidelity/frequency.

A virtual distributed actor system could potentially be used to address any potential downtime or resource waste created by unpopulated zones.

I've been playing around with some of these ideas but haven't been able to turn them into an actual implementation, so kudos to you for actually doing it!

Re: How we built an auto-scalable Minecraft server for 1000+ players

#62

Post by the_gipsy »

LennyWhiteJr wrote: post #57

I used to play on a pretty popular MMO-style server called [CivCraft]( https://www.reddit.com/r/Civcraft/ ) that unfortunately became a victim of its own success due to the large player count. It would often have 250+ players who would build these massive redstone machines and the server TPS would grind to a halt making it unplayable. After a few years it attempted a re-launch using an approach similar to to the firs…

Ah yes I immediately thought of civcraft too.

> a victim of its own success due to the large player count. It would often have 250+ players who would build these massive redstone machines

I always thought the redstone restrictions resulted in too many bots.

Re: How we built an auto-scalable Minecraft server for 1000+ players

#63

Post by Jweb_Guru »

forrestthewoods wrote: post #28
Fiahil wrote: post #20

Earlier quoted context omitted.

Is there a specific reason why Minecraft hasn’t been “adapted” from the original game to a robust design that could scale to larger worlds and player populations before your project ?

Pretty much no video game engine scales with cores. There are too many dependent systems. There's no game engine that I'm aware of that effectively uses more than a couple of cores for gameplay simulation. Distributed physics is a well understood problem with no "solution". Just different trade-offs. This solution is cool, but it's not novel or anything. Minecraft is particularly challenging because the entire world…

Video games can scale just fine with many cores, including MMOs and games like Minecraft. Veloren, for example, uses multiple cores everywhere for just about everything--most of the gameplay loop, physics, chunk meshing, world generation, realtime simulation, rendering (can be parallelized a lot further now that we've switched to wgpu), networking, and background tasks like snapshotting persistence--and it is certainly not hard to find places where parallelization yields real speedup (quite the opposite--every time we further parallelize physics it yields significant performance improvements!). There are many hard problems in video game performance, but "games just can't make good use of multiple cores" is not one of them.

If anything, I often wish we could rely on many more cores being available than actually are! It is certainly far far easier to see real performance wins with multicore than by using multiple servers, which introduce very heavy coordination costs.

Re: How we built an auto-scalable Minecraft server for 1000+ players

#65

Post by bobsmooth »

tdeck wrote: post #40

I'm always amazed by how so much Minecraft innovation is driven entirely by hobbyists. For years Minecraft was unplayable on many machines without the Optifine plugin. Even with a huge corporation behind it, Microsoft didn't build the capacity for large-scale servers, and the very large paid servers seem to use region based sharding with portals. This project is truly exciting.

Like all good things it started small and has had decades of building atop old code.

Re: How we built an auto-scalable Minecraft server for 1000+ players

#66

Post by forrestthewoods »

Jweb_Guru wrote: post #63
forrestthewoods wrote: post #28

Earlier quoted context omitted.

Pretty much no video game engine scales with cores. There are too many dependent systems. There's no game engine that I'm aware of that effectively uses more than a couple of cores for gameplay simulation. Distributed physics is a well understood problem with no "solution". Just different trade-offs. This solution is cool, but it's not novel or anything. Minecraft is particularly challenging because the entire world…

Video games can scale just fine with many cores, including MMOs and games like Minecraft. Veloren, for example, uses multiple cores everywhere for just about everything--most of the gameplay loop, physics, chunk meshing, world generation, realtime simulation, rendering (can be parallelized a lot further now that we've switched to wgpu), networking, and background tasks like snapshotting persistence--and it is certain…

> There are many hard problems in video game performance, but "games just can't make good use of multiple cores" is not one of them.

That's not quite what I said.

I'm happy you're using specs to work on an open source game. Specs and bevy and all the ECS work being done is super exciting and fun. I In the meantime there are no major games that effectively scale to, let's say, 64 cores. I don't know of anything shipped that can saturate a 12-core/24-thread Ryzen. ECS alone will not get us to that level of scaling.

Yes it's trivial to to throw audio, networking, and a few other subsystems onto separate threads. Modern games definitely leverage 4 cores. Although several of those cores will be severely under utilized.

Modern ECS designs are rapidly evolving and rapidly improving our ability to better leverage multiple cores. But we're not yet to a point where games can easily and efficiently saturate 10+ cores.

Personally I'd love to see a game like Eve Online that can effectively simulate a universe with tens of thousands of players either spread across the universe or all in one places during one giant battle.

> It is certainly far far easier to see real performance wins with multicore than by using multiple servers, which introduce very heavy coordination costs.

This is extremely true.

Re: How we built an auto-scalable Minecraft server for 1000+ players

#67

Post by z3t4 »

Being able to serve 1000+ players in a spatial "MMO" game is like the holy grail of netcode programming...

> Here's a demonstration showcasing 1000 cross-server players, this simulation is functionally identical to real player load. The server TPS never dips below 20 (perfect) and I'm running the whole thing on my laptop.

If it can run on one laptop, why does it need horizontal server scaling? :P

You don't really know where the bottle-necks are until you put 1000 actual players on the same "server".

Re: How we built an auto-scalable Minecraft server for 1000+ players

#68

Post by Jaxkr »

lwansbrough wrote: post #61

Hmm, I still think spatially allocated servers offer the best scalability. You would just have to approach it a little differently. Server boundaries could move based on where the population is via delaunay triangulation (instead of fixed boundaries), and servers could share high-importance information with their immediate bordering neighbours. (This would be recognized as ghost data on the neighbouring servers.) You…

Thanks for your comment. Spatially allocated servers was my original implementation! There’s a link to a video demo in the blog post. You are definitely correct that spatial sharding is easier and more robust.

This new approach was primarily made because I thought it would be fun and cool :P

Re: How we built an auto-scalable Minecraft server for 1000+ players

#69

Post by Jweb_Guru »

forrestthewoods wrote: post #66
Jweb_Guru wrote: post #63

Earlier quoted context omitted.

Video games can scale just fine with many cores, including MMOs and games like Minecraft. Veloren, for example, uses multiple cores everywhere for just about everything--most of the gameplay loop, physics, chunk meshing, world generation, realtime simulation, rendering (can be parallelized a lot further now that we've switched to wgpu), networking, and background tasks like snapshotting persistence--and it is certain…

> There are many hard problems in video game performance, but "games just can't make good use of multiple cores" is not one of them. That's not quite what I said. I'm happy you're using specs to work on an open source game. Specs and bevy and all the ECS work being done is super exciting and fun. I In the meantime there are no major games that effectively scale to, let's say, 64 cores. I don't know of anything shippe…

If you're just using ECS to parallelize disjoint subsystems then no, it won't get you there. But if you're using it (as Veloren is, and as I hope more people do) in a more deliberate way, to further parallelize within a system, you can indeed scale quite well to large numbers of cores. I've done some theoretical bottleneck calculations and we will still have tons of work to do with 64 cores available, if the game is written properly. We can already get decent utilization out of 32 threads at busy times, and the server was not really close to peak load in terms of player count (and we are very far from done optimizing): https://media.discordapp.net/attachments/539518074106413056/....

That's just for our server. Our clients can make use of cores in even more ways, although they have less work to do and generally have fewer cores available, and you can see Veloren taking advantage of 16 client threads with similar utilization here: https://twitter.com/sahajsarup/status/1431837669391142916.

The most important thing I want to note is that in both cases, you are not seeing tremendous imbalance between the cores most of the time. While there are definitely single-threaded bottlenecks in games, you have to be working pretty hard before they start bottlenecking the workload! Instead, we are just suffering from a combination of general inefficiency and lack of work to do.

So no, I'm going to push back against this notion that multicore scaling for games is some sort of crazy intractable problem. It's not. Like any other kind of parallel scaling, it's trivial in some places, more challenging in others, and depends a lot on your workload (including you actually having enough work to saturate the cores in the first place!). But there's nothing special about games here.

Re: How we built an auto-scalable Minecraft server for 1000+ players

#70

Post by MuffinFlavored »

Jaxkr wrote: post #18
cdev_gl wrote: post #12

Earlier quoted context omitted.

I rent a small Linode to run a minecraft server for my extended family. Sometimes it struggles to keep up with just a half dozen of us, and when I tried swapping from a spigot based system to a forge one so the kids could have mods it was basically unplayable. I can't imagine scaling out to thousands of users, this was some amazing work. While developing this did you come up with any best practice recommendations for…

Your single-core performance is the most important thing for a Minecraft server, especially for Forge mods. For vanilla, I’d try Paper (a high performance spigot fork) and see if you still have problems. If you’re lagging with 6 people while running Paper, you simply need a better CPU.

Tangentially related, I had to move a web app off of DigitalOcean because the CPU performance was terrible compared to what you get for $80/mo from a dedicated server from OVH. I know Linode and DigitalOcean compete. Get a OVH dedicated server, try the CPU passmark bench or whatever and compare. AWS + Linode + DigitalOcean are all VPS and pretty slow.
Post reply on HN