Anyone else think this is a sad use of both human and computing resources? Technically interesting, yes, but all of that for playing freaking farmville?
Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
31–40 of 53 posts
Re: Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
#321000 servers per week number is not just realistic. I will like to believe the author misspoke otherwise it will take more that Zynga employees just to configure 1000 servers in a week time.
Umm... With EC2, which is what they often use, powering up and configuring 1000 machines can take 10 minutes or so, give or take a ramped boot. :-)
It makes you wonder how many servers Amazon adds per week..
Re: Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
#33This makes me feel a little better. We created a multi-player rpg for mobile phones at startup weekend a few year ago. It actually did ok and it was growing pretty quickly. I hosted it on AppEngine to start and the resources just started going sky high. Players were initiating an attack at least once per second (not to mention all of the other server calls like messaging, weapon purchasing, scanning to see who is in…
That'd be great if you could elaborate a bit on how appengine wasn't fit or too expensive for your use case? Was performance an issue, or just costs? Thanks.
Just cost.
There were some issues with a high number of writes, but it has got better.
The unscheduled downtime also killed us.
But really, it was just the sheer amount of traffic. Real-time multi-player games that people have with them all the time can just create a ton of requests.
Re: Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
#34Earlier quoted context omitted.
That'd be great if you could elaborate a bit on how appengine wasn't fit or too expensive for your use case? Was performance an issue, or just costs? Thanks.
Just cost. There were some issues with a high number of writes, but it has got better. The unscheduled downtime also killed us. But really, it was just the sheer amount of traffic. Real-time multi-player games that people have with them all the time can just create a ton of requests.
Thanks for the reply. Do you think another hosting environment (factoring in the administration costs and all) would have been cheaper?
Re: Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
#35Earlier quoted context omitted.
Downvoted for not thinking through the numbers. When Zynga launches a new game, they can get over 1M new DAILY players on that game within the first week. [1] For their games, it seems that 7% of daily players are online at any given time. [2] So within the first week of launch, you have, about 100,000 people playing, simultaneously. Let's say the client is sending 5 requests per second to the game server. And let's…
100'000 people playing simultaneously adds up to say about 10'000 requests per second. Farmville-like games are not exactly rocket science - most of those requests should be processed sub-50ms total cpu-actual-usage time. that's 200 requests/s per server. Which adds up to 50 servers. Let's double that to account for db load too, and we're still an order of magnitude away from 1000 servers.
So each player generates a request every 10 seconds? Sorry, but that's way off. Every time they click, at least, generates a request. And then there's the famous Zynga tracking. There's no way it averages to less than a request a second.
Re: Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
#36Apparently this are 1,000 ec2 instances not real servers. This changes a lot of things.
Btw how is it possible that 10 or 20 years ago using megaflops or MIPS (or other sane measures of computer power) was normal and now we are here talking about "servers"? :)
Edit: I'm searching around the internet if there is some new general measure of computer power that is able to reflect the real-world performance in non scientific applications. No luck so far.
Re: Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
#37Earlier quoted context omitted.
An average of 5 requests per second from a casual game's client is a sign of a poor design. Realistically you very close to 1 user action = 1 request. Now, a top tier starcraft micromanager may average pull 300 actions per (ed: minute), but an average user should be somewhere around 1 action per second to 6 actions a minute. PS: The key to pulling this off is running the same simulation on the client using the same r…
> 300 actions per second You mean per minute, right? Either that or I'm way worse at SCII than I thought.
Ops, fixed.
Re: Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
#38Here's how this adds up to me: - 215M users. - Up to 1000 servers per week. - Say, they've been doing this for 30 weeks. - So they need 1 server for every 7167 users? Is this normal for mmorpg type games? - Are 215M people playing around the clock? I'm just a little baffled by this number.
We develop casual MMOs (think Club Penguin). These are not nearly as complex as a WOW, much closer to Farmville. In our case, like most MMOs, nearly all our messages go back and forth over persistent socket connections, not HTTP requests. Firewalls cause problems and often block persistent connections, but we can get a much much better through put.
For example, we developed a product for Turner on top of our platform. Take a worst case scenario - everyone in the game moves using arrow keys, and they're all moving. A typical room might have 25 people, each broadcasting their location roughly every 0.5 seconds. This location update is broadcast to all 24 other people in the room. So each person generates 2 messages into the server per second, but results in 48 broadcast messages per second. Of course there are 25 people in the room, all of whom are moving. So we have 2 x 25 = 50 messages into the server per second, and 48 x 25 = 1200 out going messages per second.
A typical node of our platform might support 4000 players. If each room supports 25 people then we have 160 rooms. Every second the server needs to broadcast 1200 x 160 = 192,000 or nearly 200,000 messages, and receives close to 8000 messages.
Movement packets are easy. We can route these through the system quickly, but then think about the "slow" requests, such as writes to the game state persistence store (Mongo DB) or slower still, writes to the SQL database.
With 8000 incoming messages, and a typical 64 thread processing pool each thread needs to churn through 125 messages per second. Or one message every 0.008 seconds. Those slow SQL writes might take 0.5 second, during which time 0.5/0.008 = 62 messages have backed up.
Many of our games (and im inferring Zynga's) store huge amounts of persistence / state data. Where has the player been, whats completed, timers, friends, etc, etc. Much more than you'd imagine - and thats not including the analytics data. These slower writes very quickly add up and reduce throughput.
Of course there are lots of things to optimise the number of messages, but this starts to give an idea for how quickly messages back up in a MMO type environment.
If you've bothered to read this far you might be interested in a short blog post I wrote about our casual MMO architecture http://dubitplatform.com/blog/2010/3/4/under-the-hood-dubit-...
Re: Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
#39Here's how this adds up to me: - 215M users. - Up to 1000 servers per week. - Say, they've been doing this for 30 weeks. - So they need 1 server for every 7167 users? Is this normal for mmorpg type games? - Are 215M people playing around the clock? I'm just a little baffled by this number.
There are a couple of things:
1. The servers aren't all application servers. There's load balancers, memcache servers, lossy memcache servers, etc.
2. It's the cloud. It includes large instances, HA-Proxies, memcache servers, etc. 3. There's very high write characteristics for social gaming's data store, but the read is similar to normal web application.
4. He took an aggregate server #, and generalized it over a period of 30 weeks. It's directionally accurate, but I wouldn't take it literally.
Re: Zynga Moves 1 Petabyte Of Data Daily; Adds 1,000 Servers A Week
#401000 new servers a week is a ridiculous number for this sort of company. Their code must be really, really bad. Anyone who works there care to chime in? Is it all written in ColdFusion or something?
Is it all written in ColdFusion or something? http://gigaom.com/2010/06/08/how-zynga-survived-farmville/ Zynga uses Apache PHP on the front end, memcached for active user play and MySQL on the back end. It uses memcached to store key value pairs to deal with active user play during sessions and then later writes it to disk.
I'm obviously asking the wrong person, but if they use memcached as a temporary datastore until another process can come along later and write the data to disk, what do they do if a memcache node fails?
Or would a node fail rare enough for the data loss to not matter much?