Openbsd folks used lpq to queue mp3 playlist. Cups is also an option and you get full stack of goodies built in.
http://patrick.wagstrom.net/weblog/2003/05/23/lpdforfunandmp... http://rendermania.com/building-a-renderfarm-with-cups/
5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
41–50 of 66 posts
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#42I've been met with looks of disgust for using a filesystem to implement a queue, but I feel it's unjustified. A modern unix filesystem is surprisingly well suited to this task: You get atomicity "for free", inotify allows it to be interrupt driven rather than polled, it inherently supports multiple processes (thus different parts of the system can be implemented in different languages), there's no need for locking as…
I recently read through news.arc (the source to Hacker News itself) and was dumbfounded by how such a simple, file-system backed system was able to cleanly and performantly handle many of the use cases of a document store or key value store. Are there any good resources on the DOs and DONOTs of building apps in this "Hey.... dummy.. Just use the file system!" -style?
One big DONOT is: Don't do this if you need more than one physical host to be processing the jobs at the same time. Resiliency is hard to get right with shared filesystems.
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#43Earlier quoted context omitted.
But 0MQ can lose items too if the queue fills, how does this help? I can see adding a queue with persistence would work...
Unless there are so many items per sec that your persistence can't keep up. Wouldn't this kind of create the same situation: You cant accept all the new items and have to throw some away. Only, now everything is slower. A lot slower. Ok, the first scenario is caused by the workers being too slow, so it's not exactly the same :)
Yes, persistent queues have issues, which is why 0MQ exists. But using a non persistent queue to deal with overflow just delays the problem, which was why I asked...
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#44Earlier quoted context omitted.
But 0MQ can lose items too if the queue fills, how does this help? I can see adding a queue with persistence would work...
Yeah, that's true. But my environment is such that any one of 100 or so app servers has a significantly lower chance of running out of memory than the Redis server does. The 0MQ high water mark is set high enough so that it's virtually impossible not to fix a broken DB by the time messages on the client side create an OOM condition being queued in memory.
high enough so that it's virtually impossible not to fix ... by the time
I'll add that to my "famous last words" fortune cookies.
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#45Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#46Earlier quoted context omitted.
MySQL != MyISAM. Check out InnoDB :)
InnoDB isn't too hot on transactions either. Google for "InnoDB deadlock".
I've googled, but am seeing the same kind of results that I see when I try "oracle deadlock", "postgresql deadlock" and "db2 deadlock."
Still being relatively new to InnoDB, what particular deadlock dangers make it more troublesome than other DBMS engines?
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#47> Instead of SELECT FOR UPDATE followed by UPDATE, just UPDATE with a LIMIT, and then see if any rows were affected
Should be noted, this is not necessarily a good solution: a concurrent consumer, which may be another incarnation of a given script running with a lag, may hijack the queue element locked this way; as a result you may end up having two or more incarnations of the consumer handling the same queue element.
The most universal approach to DB queues is to assign each consumer process a unique ID which it should use for locking queue elements in their UPDATE ... LIMIT 1.
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#48And here is one explicit way to use MySQL as a queue: https://www.pingbrigade.com/blog/entry/selector-workers-reco...
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#49I've been met with looks of disgust for using a filesystem to implement a queue, but I feel it's unjustified. A modern unix filesystem is surprisingly well suited to this task: You get atomicity "for free", inotify allows it to be interrupt driven rather than polled, it inherently supports multiple processes (thus different parts of the system can be implemented in different languages), there's no need for locking as…
Agreed++. I've done similar filesystem queues and have been told by corworkers "Yuck! That needs to be in a database!" ... so I ask why ... and the answer is "Because that's what databases are for!" Inevitably these cowrokers inherit the project, database every aspect of it, and then the app promptly collapses into a steady stream of downtime alerts. Yes, that's what databases are for: keeping DBA's gainfully employed.
Re: 5 subtle ways you’re using MySQL as a queue, and why it’ll bite you
#50My project uses couchdb for task queuing:
https://github.com/codehero/scheddesk
Still kind of alpha, but working for my purposes.