Disque 1.0 RC1 is out

Does a proper "queing" project, that is messaging independant exists ? That would open the door to a standardization of the queing semantics and API.

Like, if I want to use crossbar.io (with it's WAMP protocol) for the messaging part, but need task queue, what are my options ?

* Has a well-defined lifetime — unstarted, running, paused, completed successfully, or failed;

* Is executed from some kind of parameterized "job specification" that describes its inputs and desired behavior;

* Has state data (e.g., completion % progress, log output, metrics, transactional continutation state for pausing/retrying);

* Can be scheduled on multiple nodes;

* Can be created and scheduled multiple times;

* Can be scheduled by priority;

* Can have its resources (RAM, CPU, I/O) constrained.

The thing is, queues are terrible at this. People continue to abuse messaging systems as a scheduler, and it always sucks.

For example, consider acking: For a long-running tasks that might take hours, do you really want to hold up the broker's internal lock for all that time? What if you restart the broker — the task is still running, now you decoupled its running state from the queue. Similarly, queues are terrible at finding out what's waiting, or what has happened. By using the queue as job state, you are coupling scheduling with execution, which is just wrong.

A queue (one that supports priorities, mind you) is, however, excellent for management commands: "Start job X", "pause job X" and so on. But you will want to maintain execution state separately from the queue. A relational database is quite good at this, though one could use a NoSQL databases, too. First, create a row representing the job: {id="job1", state="new", type="import_stuff", params="..."}. Then push a queue message {commmand="start_job", job_id="job1"}. Run a queue consumer or ten on each node. Each consumer acks the message, finds the job in the table and starts it. One could run it as a child process, or in a Docker container. One could of course use an orchestration service (like Mesos/Marathon) to run it, and monitor the service for container events and update the job table appropriately.

I would want to take it further and allow the job to update itself through API, so that it could publish metrics and log messages, as well as maintain whatever data is useful snapshotting. For example, a long-running task to import data into a database would benefit from maintaining a cursor so that you could pause/kill the job and later resume it at the point where it was.

RabbitMQ does not have a good strategy for recovering from partitions, which happens when a node is unable to talk to its peers. Partitions can occur not just from actual network hiccups but also simply due to high CPU or I/O load or benign VM migrations.

The underlying cause is that RabbitMQ is not multi-master by default. A queue is owned by a specific node, and if you have a partitioned cluster, that queue (and related objects such as exchanges and bindings) will simply disappear from other nodes.

You can patch this deficiency by enabling "high availability" (HA), which is their term for mirrored queues. Each queue will get a designated master, and be replicated to other nodes automatically. If a partition happens, the nodes elect a node to become a new master for a mirrored queue.

Unfortunately, this opens the cluster up to conflicts. Let's say you have two nodes, A and B, with one queue, Q. You experience brief hiccup causing a partition. A and B will both assume the role of master for Q, because RabbitMQ has no quorum support. Therefore they will continue to accept messages from apps. The hiccup passes, and now both nodes see each other again. Meanwhile, apps had sent messages to both A and B, causing Q to diverge into Q^1 and Q^2.

However, RabbitMQ has no way to consolidate the two versions into a single queue. To fix this situation, either you need to reconstruct the queue manually (usually impossible from an application's point of view), or wipe it (hardly a solution in a production envirionment), or simply have RabbitMQ automatically pick a winning master and discard the other master(s). The latter strategy is called "autoheal", and will automatically pick the master which has the most messages. The previous master(s) are wiped and become slaves. This is coincidentally the only mode in which RabbitMQ can continue to run after a partition without manual intervention. Without autoheal, RabbitMQ will become unusable.

In practice, recovery has proved flaky for us. Nodes often stay partitioned even after they should be able to see each other. We have also encountered a lot of bugs — for example, bindings or exchanges disappear on some nodes but not on others, or queues are inexplicably lost, or nodes otherwise just misbehave. We're on a cloud provider which is otherwise rock solid; of all the software (databases etc.) we employ in our clusters, RabbitMQ is the only one that misbehaves. I should add that the last few minor versions have increased stability considerably, though the fundamental clustering design issue remains.

[1] https://aphyr.com/posts/315-jepsen-rabbitmq