Facebook's Memcached Multiget Hole: More machines = More Capacity

If you store and retrieve an object only to server # (object.id % numservers), you don't face this issue. (Obviously you could use a better hash than %.)

Re: Facebook's Memcached Multiget Hole: More machines = More Capacity

#5

Post by sstrudeau » Mon, Oct 26, 2009, 5:55 PM UTC

compay wrote: post #3

bantic wrote: post #2

the title should be != More Capacity

Yes, as in - this is not sarcasm; the actual article title is: "Facebook's Memcached Multiget Hole: More machines != More Capacity"

Right.

Though, to argue with the article's title -- while a straightforward addition of more memcached nodes did not solve their capacity problem, the strategy of replicating nodes & load balancing read requests between them appears to be a solution to this particular capacity problem; so more machines = more capacity, so long as they are organized appropriately.

Re: Facebook's Memcached Multiget Hole: More machines = More Capacity

#6

Post by almost » Mon, Oct 26, 2009, 6:33 PM UTC

yummyfajitas wrote: post #4

Being a little smarter on the client side can help a lot here. If you store and retrieve an object only to server # (object.id % numservers), you don't face this issue. (Obviously you could use a better hash than %.)

This seems so obvious there must be a good reason why they're not doing it. Anyone know?

Re: Facebook's Memcached Multiget Hole: More machines = More Capacity

#7

Post by sstrudeau » Mon, Oct 26, 2009, 6:34 PM UTC

yummyfajitas wrote: post #4

Being a little smarter on the client side can help a lot here. If you store and retrieve an object only to server # (object.id % numservers), you don't face this issue. (Obviously you could use a better hash than %.)

This would help to distribute requests for individual objects, but doesn't help for multi-get requests. The problem is adding more nodes increases the number of actual requests a multi-get call needs to make (assuming you're asking for a sufficient # of objects they are likely to be distributed across all nodes). This decreases the # of keys requested per node but increases the total number of requests to the cluster. Because the bound was on throughput of requests (bound by CPU), minimizing the number of keys per request to a node doesn't help.

The proposed solution is to instead replicate nodes and load balance read requests. In this case, this doubles your read capacity, though you must write twice (or N times depending on your replication level).

Re: Facebook's Memcached Multiget Hole: More machines = More Capacity

#8

Post by confusedcitizen » Mon, Oct 26, 2009, 6:38 PM UTC

yummyfajitas wrote: post #4

Being a little smarter on the client side can help a lot here. If you store and retrieve an object only to server # (object.id % numservers), you don't face this issue. (Obviously you could use a better hash than %.)

Well, this is the underlying logic indeed. What the author mentions is seeing 50 requests for 100 friends, where these friends might be divided into 3 servers. So, the keys are hashed to servers, i.e., if there is only 1 multi-get request, each server would see only 1 request. However, if there are 50 multi-get requests, each server would see 50 requests irrespective of the number of keys (friends) it caters to.

Re: Facebook's Memcached Multiget Hole: More machines = More Capacity

#9

Post by learnalist » Mon, Oct 26, 2009, 7:24 PM UTC

yummyfajitas wrote: post #4

Being a little smarter on the client side can help a lot here. If you store and retrieve an object only to server # (object.id % numservers), you don't face this issue. (Obviously you could use a better hash than %.)

It might be obvious, BUT i know my knowledge doesnt include working with 28 terabtyes of data. Just reading it in the article is daunting!

I wonder what sort of sandbox they have to play on. They also probably lack the ability to Google or stackoverflow the problem.

Re: Facebook's Memcached Multiget Hole: More machines = More Capacity

#10

Post by lucifer » Mon, Oct 26, 2009, 7:42 PM UTC

sstrudeau wrote: post #5

compay wrote: post #3

Earlier quoted context omitted.

Yes, as in - this is not sarcasm; the actual article title is: "Facebook's Memcached Multiget Hole: More machines != More Capacity"

Right. Though, to argue with the article's title -- while a straightforward addition of more memcached nodes did not solve their capacity problem, the strategy of replicating nodes & load balancing read requests between them appears to be a solution to this particular capacity problem; so more machines = more capacity, so long as they are organized appropriately.

Actually if you rewrite the equation you can see that:

   more capacity != more machines
   more capacity = more machines + load distribution strategy