5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
1–10 of 66 posts
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#2This is why Redis / *SQL is my favored stack. It just covers so many bases, you get things like safe queuing, caching, pub/sub, and weird high-performance low-durability cases from Redis, and great, safe relational support from SQL.
For best results, it's good to have at least two redis servers, one with snapshotting as a cache (fast, less durable), one with 1 second Append only files (still fast, but slower) for data you care more about.
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#3Is the page's rendering totally busted for anyone else? Chrome on Ubuntu.
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#4Percona always seems to have good articles.
I think, as he said, everyone shouldn't run out and replace a mysql job queue for their wordpress blog. In a great many cases it doesn't matter.
I also like how he never said "Don't use mysql as a queuing system" but "be careful of these things". I've used mysql as a queuing system, and it works fine. I looked at replacing it with a different database, but in that situation it was not worth the investment.
Signaling mysql + archiving performed work + no locks that lock more than the exact row that's being updated (and also avoiding concurrent workers acting on the same task) will take a mysql backed queuing system far. I've set up a system that processes well over 5,000 tasks / day using it.
Do I think everyone should use mysql as their queuing backend? No. People should probably use a queuing library, with persistence to a database (redis?) enabled for critical tasks. Of course, as the article said, be careful about the choice of backends.
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#5Is the page's rendering totally busted for anyone else? Chrome on Ubuntu.
Fine for me with Chromium 13.0.782.220 on Arch.
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#6This is why Redis / *SQL is my favored stack. It just covers so many bases, you get things like safe queuing, caching, pub/sub, and weird high-performance low-durability cases from Redis, and great, safe relational support from SQL. For best results, it's good to have at least two redis servers, one with snapshotting as a cache (fast, less durable), one with 1 second Append only files (still fast, but slower) for dat…
What happens when the queues back up and Redis runs out of RAM?
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#7Is the page's rendering totally busted for anyone else? Chrome on Ubuntu.
Normal on Chromium 13.0.782.215 on Ubuntu 11.04
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#8I'm just going to count killing a SLEEP(100000) query as a means of signalling a worker as the something new I learned today. I'm not sure I've ever written anything where implementing that would have had any real impact, but it's filed away for the future.
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#9Is the page's rendering totally busted for anyone else? Chrome on Ubuntu.
Was broken on first load for me, Ubuntu 11.04 Chrome 13.0.782.220.
However when I reloaded the page it fixed itself. In fact as the page reloads I can see the text layout first breaking and then immediately fixing itself...
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#10Queues in the DB are so common that in MSSQL they made it a first class feature: SQL Server Service Broker. Using it is an XML and T-SQL nightmare, but since it guarantees in-order, only once delivery, and supports routing and in-DB worker activation, you can build some really robust and powerful stuff with it.
MySpace used it to keep their partitioned databases in sync: http://www.microsoft.com/casestudies/Case_Study_Detail.aspx?...