Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue

Asynchronous Processing in Web Apps, Part 1: A Database Is Not a Queue
blog.gomiso.com

I am a message queue skeptic. Not that they should never be used, but rather a general feeling that complex, dedicated message queue software is often used for engineering problems between two or three orders of magnitudes too small before they deliver value. And, for most projects, queue replacement is not too difficult, especially if one's use of a database-backed queue is relatively naive.

As such, I will -- with an open mind -- suggest that general purpose database management software masquerading as queues is not an outright antipattern, and the times to use them in this way is probably more commonly seen than the opposite, where a dedicated queue delivers clear value.

Here are my main reasons for thinking that, which almost entirely have something to do with being able to address one's queue and one's other data in the same transaction:

* Performance: a pedantically unsound but basically reasonable rule of thumb (as experienced by me) would suggest that one needs to be processing somewhere between hundreds or thousands of messages per second with at least fifty (and maybe up to a hundred or two) parallel executors processing or emitting messages before there are performance issues where the lower constants of a dedicated queuing package become attractive. Below this kind of throughput, one starts to experience some more pain than gain.

* Correctness: Most database + queue integrations do a lousy job of what is effectively two-phase commit between different data storage instances (so that would include two+ RDBMSes, even if they are of the same kind, e.g. 2xPostgreSQL), and frequently the queue has to be counted among these (exception: when the queue contents can be lost/can be rebuilt/is idempotent at all times). Systems do a lousy job of making this work because it's pretty finicky to do a good job in many situations, i.e. expensive and time consuming.

* Constants, when dealing with other systems: When one does do a good job and has interesting requirements in the 'correctness' case, it often means doing forms of two-phase commit, whether explicitly supported by the system (e.g. PREPARE TRANSACTION) or a spiritual equivalent via carefully thought out state machines. In principle these could be relatively cheap, but typically to avoid complexity more expensive approaches are employed, such as a couple extra UPDATE requests to poke at some home-grown state machine.

Also, my experience indicates that as systems evolve, there will be inevitable bugs in these state machines that, by nature, span systems. Be vigilant and make sure you get more value than pain, and try to avoid having too many of them.

* HA is still hard: clustering is generally in principle possible, but make sure you read the fine print. For example, many people use Redis as a queue, but it is not really unlike any other monolithic database most generally -- its main draw as a vanilla queue is good execution-time constants. The same could be said of Apache ActiveMQ in its least byzantine configuration. One might think that one would get a lot of leverage 'for free' given the simpler semantics of queues vs the diversity of access methods in most general purpose databases, but so far I have not seen that to be the case, for the very good reason that a lot of people expect a lot of reliability out of their queues (no less than the transactional nature of some databases), and doing that is either most natural in a monolithic system or slow, or complicated, or both in a multi-master distributed system.

All in all, if you think you need dedicated queuing software to send a few dozen emails a second (that's a lot of email for most people!), think twice: it might still be a good idea, but brace yourself for these pitfalls or convince yourself that they probably mostly don't apply to you.

Traditionally, but not necessarily; PostgreSQL supports the LISTEN and NOTIFY commands for asynchronous notifications, without polling.