Code rant: The Database As Queue Anti-Pattern
mikehadlow.blogspot.se
Code rant: The Database As Queue Anti-Pattern
1–10 of 15 posts
Re: Code rant: The Database As Queue Anti-Pattern
#2Using databases as a queue is not an anti-pattern (delayed_job is an example).
A full blown message queue system is often overkill; it adds another system that you have to learn. (And frankly, queues are often backed by databases anyway)
That said, it's best to avoid having application state in the queue, it should be generic. This makes it more scalable and flexible.
Re: Code rant: The Database As Queue Anti-Pattern
#3Using databases as a queue is not an anti-pattern (delayed_job is an example). A full blown message queue system is often overkill; it adds another system that you have to learn. (And frankly, queues are often backed by databases anyway) That said, it's best to avoid having application state in the queue, it should be generic. This makes it more scalable and flexible.
Also, if your using the DB as a queue index on the status element. Inserts are still low overhead and the DB can handle poling that table 10,000 time a second without issue.
Edit: Just don't pole the 'finished' status as that can have a lot of elements in it.
Re: Code rant: The Database As Queue Anti-Pattern
#4A lot of that post is specific to SQL Server. I used a MySQL table as a queue as part of Signal Spam (https://www.signal-spam.fr/) and it worked flawlessly. Polling was at 1 minute intervals and queue entries that were completed were simply deleted.
Re: Code rant: The Database As Queue Anti-Pattern
#5Using databases as a queue is not an anti-pattern (delayed_job is an example). A full blown message queue system is often overkill; it adds another system that you have to learn. (And frankly, queues are often backed by databases anyway) That said, it's best to avoid having application state in the queue, it should be generic. This makes it more scalable and flexible.
Also, if your using the DB as a queue index on the status element. Inserts are still low overhead and the DB can handle poling that table 10,000 time a second without issue. Edit: Just don't pole the 'finished' status as that can have a lot of elements in it.
A lot of databases can also provide partial indices, which are only maintained and searched when a WHERE clause matches (example: in this case, WHERE status != 'filtered' would be a good choice).
Re: Code rant: The Database As Queue Anti-Pattern
#6But no-one polls a database anymore; they all have a callback mechanism an app can just subscribe to.
Re: Code rant: The Database As Queue Anti-Pattern
#7I use MySql+cron as a simple message queue every now and again--it's great for jobs like sending confirmation emails or registering new users for a mailing list. ie. Tasks that make synchronous calls over the network and can degrade the speed of important processes (registration, etc). You just have to make sure that these tasks are going to be infrequent and not prone to spikes that might overwhelm the server.
Re: Code rant: The Database As Queue Anti-Pattern
#8A lot of that post is specific to SQL Server. I used a MySQL table as a queue as part of Signal Spam ( https://www.signal-spam.fr/ ) and it worked flawlessly. Polling was at 1 minute intervals and queue entries that were completed were simply deleted.
That particular use-case is such an anti-pattern in SQL Server ( because SQL Server isn't or hasn't until recently been MVCC ) that Microsoft added Service Broker to the product.
It isn't nearly the anti-pattern, at least for many loads, in an MVCC database engine.
Re: Code rant: The Database As Queue Anti-Pattern
#9The database is not the world's most efficient way to do this. However if it isn't a bottleneck, who cares? It is one less dependency to maintain, understand, etc.
We're looking to create working solutions, not works of art. Seeking the perfect component for every piece is often simply not worthwhile.
Re: Code rant: The Database As Queue Anti-Pattern
#10I find it ironic that he writes this as a rant, which is an anti-pattern for influencing people. One of his commenters hints at how it might have been better presented: introduce messaging systems, explain how they solve the problem, and describe their advantages over the database. For further discussion on why a rant is probably an anti-pattern, see Dale Carnegie's "How to Win Friends and Influence People."