Behind a Backend-as-a-Service Provider: The How and Why of Our Architecture.

We actually tested a few designs using m2 and 0mq during our R&D phase. 0mq's push/pull sockets provide the same "take" behavior as the Redis RPUSH/BLPOP, so we definitely could have used m2 as the HTTP end of a similar architecture to what we use now.

One of the considerations that led to the choice of Redis was the transparency of the queueing and dequeueing. The messages in the queue are easily inspected, and the Redis MONITOR command helps greatly in debugging.

More important from a design perspective was our desire to hide the HTTP specifics from the workers. m2 pushes a JSON or tnetstring representation of the HTTP request to its workers, but we want the task we send to the workers to be generalized, stripped of information that is only meaningful for HTTP. We also want to classify the task by resource type and requested action, which allows us to use multiple queues. Multiple queues allows us to implement workers on an ad hoc basis.

M2 could work here if our request classification only depended on the URL. But that is a limitation we are not willing to accept. Request headers can be very useful in dispatching, especially those related to content negotiation (Accept, Content-Type, etc.)

There is an interesting hybrid approach using mongrel2: write one or more m2 handlers that perform the same function as our node.js dispatchers. I.e. m2 sends the JSON-formatted request to an m2 handler that deserializes it, removes the HTTP dressing, classifies the request according to type and action, then queues a task in the appropriate queue. A worker takes the task, does its own little thing, and sends the result to an m2 handler that knows how to re-clothe the result as an HTTP response and deliver it to the m2 cluster.

Regarding the question about queue behavior across replicated Redises:

I do not know for certain, but I do certainly hope it is not possible for an item in a Redis list to be popped by more than one client, no matter how the replication is configured.

With our architecture, we could relieve at least some of the strain on the task/result messaging system by using a cluster of Redis servers for the task queues. Each queue server might have its own cadre of workers listening for tasks. The return trip (getting a result from a worker back to the HTTP front end) is a little trickier, because it matters which HTTP server is holding open the HTTP connection. You could use PUB/SUB (which I believe is how m2 currently does it), or each HTTP server could be popping results from its own result queue.

When using a single Redis server, the only hard limitation we have seen with using the BLPOP operation is the number of client connections Redis can keep open. In case it's not clear, the BLPOP is blocking for the client, not the server.

Do you do anything special to make the queue which your workers are BLPOPing durable?

Is there a reason you didn't use Redis pub/sub? Seems like the perfect use case.