How Facebook deals with memcache consistency in multiple datacenters
1–10 of 21 posts
Re: How Facebook deals with memcache consistency in multiple datacenters
#2note to self: don't start companies that require multiple synchronized data centers. :-)
Re: How Facebook deals with memcache consistency in multiple datacenters
#3note to self: don't start companies that require multiple synchronized data centers. :-)
While there's a ton of talk about scaling on here, it isn't needed by the vast majority of people and even Facebook's way isn't that complicated when you get down to it.
Basically, the only change that needs to be done is that you have MySQL trigger the delete from memcached on replication. All your slave database gets an updated record for you, saves it in its tables and then hits memcached with a delete command for that key. While it does mean diving into the MySQL code, when you get to the size at which you need such a feature, you can just hire a MySQL expert.
Re: How Facebook deals with memcache consistency in multiple datacenters
#4Interesting use of the 20-second cookie. What happens when they scale to data centers in Europe, Asia, and South America? Does it become a arbitrary-second cookie depending on where the user is?
Nonetheless, I can appreciate how a simple approach solves the problem. If these are the most dramatic scaling issues that facebook faces, then this provides strong support to the argument that you shouldn't focus too much on scaling and optimization when first building your site. There are probably better uses of your early time.
Re: How Facebook deals with memcache consistency in multiple datacenters
#5Interesting use of the 20-second cookie. What happens when they scale to data centers in Europe, Asia, and South America? Does it become a arbitrary-second cookie depending on where the user is? Nonetheless, I can appreciate how a simple approach solves the problem. If these are the most dramatic scaling issues that facebook faces, then this provides strong support to the argument that you shouldn't focus too much on…
I'd guess that they'd need to partition their data somehow by then, so the Europe DB is master for European users and slave for the rest, and so on.
Re: How Facebook deals with memcache consistency in multiple datacenters
#6I've always been curious why someone hasn't (or why I haven't heard of someone) prototyping an improvement to MySQL's query cache to invalidate the cache less frequently and get more dependable usage. Currently it invalidates when any table referenced by the query changes. It seems like you could improve this algorithm without too much trouble. For example, it feels like if the query was located solely through indexes, it should be able to do cache invalidation on an update to those index hits rather than on the entire table.
I'll have to look through the code one day and see why this makes no sense.
Re: How Facebook deals with memcache consistency in multiple datacenters
#7There's another easier way built into memcached:
When you issue a delete you can specify "the amount of time the client wishes the server to refuse 'add' and 'replace' commands"
Maybe they couldn't afford to increase the read load so much but this seems easier than rewriting mysql's replication engine.
Re: How Facebook deals with memcache consistency in multiple datacenters
#8[deleted]
Re: How Facebook deals with memcache consistency in multiple datacenters
#9There's another easier way built into memcached: When you issue a delete you can specify "the amount of time the client wishes the server to refuse 'add' and 'replace' commands" Maybe they couldn't afford to increase the read load so much but this seems easier than rewriting mysql's replication engine.
[deleted]
Re: How Facebook deals with memcache consistency in multiple datacenters
#10note to self: don't start companies that require multiple synchronized data centers. :-)
Actually, these are really interesting problems. Multi-homing isn't an issue for most startups though (not that this is true multi-homing since their second DC is read-only, meaning that they can't really survive the loss of the primary).