It would be interesting to hear the rationale behind using this over something like RabbitMQ, which has its own storage layer, as well as the queue aspect.
A lightweight, high-performance, language-independent job queue system
11–20 of 41 posts
Re: A lightweight, high-performance, language-independent job queue system
#12The title fits perfectly the description of ZeroMQ[1]. Except that I actually trust ZMQ to deliver on the promise.
[update] Oh, it's "job queue system", not a generic "queue system". So it's not quite a perfect fit, I guess :)
Re: A lightweight, high-performance, language-independent job queue system
#13The title fits perfectly the description of ZeroMQ[1]. Except that I actually trust ZMQ to deliver on the promise. [update] Oh, it's "job queue system", not a generic "queue system". So it's not quite a perfect fit, I guess :) [1] http://zeromq.org/
Yeah, a better example would be https://github.com/zeromq/malamute
Re: A lightweight, high-performance, language-independent job queue system
#14Honest question, how is using MySQL lightweight? Many job queues I've seen use e.g. Redis
If the requirement is for persistence, i.e. no data loss if the queue process dies, then Redis won't fit. EDIT: TIL Redis has the option to turn on fsync-to-disk on every write. Probably not what people are thinking of when they suggest Redis as lightweight.
Redis has persistence if you need it. But only if.
Re: A lightweight, high-performance, language-independent job queue system
#15Honest question, how is using MySQL lightweight? Many job queues I've seen use e.g. Redis
If the requirement is for persistence, i.e. no data loss if the queue process dies, then Redis won't fit. EDIT: TIL Redis has the option to turn on fsync-to-disk on every write. Probably not what people are thinking of when they suggest Redis as lightweight.
[deleted]
Re: A lightweight, high-performance, language-independent job queue system
#16Interesting, but I'd like to know why there is a polling_interval? Considering there is a master server for each dispatch queue, one could expect a design where polling is unnecessary.
Re: A lightweight, high-performance, language-independent job queue system
#17orphan_jobs.sql:
SELECT job_id FROM `{{.JobQueue}}`
WHERE status = 'grabbed' AND grabber_id != CONNECTION_ID()
LIMIT 1000
can it break when MySQL's thread ID is reused, and some new client will get the same CONNECTION_ID as a previous failed worker?
Re: A lightweight, high-performance, language-independent job queue system
#18> It is built on top of RDBMS (MySQL), Can I hear more about the rationale behind this?
I'm not involved in the project, but you do this if you want the same robustness as an rdbms, e.g. if a client gets an acknowledgement back from the queue then the data is definitely captured and won't be lost if the queue process then crashes.
I agree that it makes sense in certain circumstances and definitely depends on requirements (not only transactional enqueues which you mentioned, but also a lot of times it's a need for O(lg n) lookup/update/deletion of jobs in flight), and I've also seen a few MySQL-backed queues in production that can handle fairly high throughputs. I'm particularly and mostly interested why this project does it.
Re: A lightweight, high-performance, language-independent job queue system
#19The title fits perfectly the description of ZeroMQ[1]. Except that I actually trust ZMQ to deliver on the promise. [update] Oh, it's "job queue system", not a generic "queue system". So it's not quite a perfect fit, I guess :) [1] http://zeromq.org/
Yeah, a better example would be https://github.com/zeromq/malamute
Well, my original message was a bit snarky, but what I meant was that, at the very least, this project advertises the wrong features. I mean come on, "lightweight" and "high-performance"? Maybe advertise safety properties, resilience, I don't know... but the way it is built, it can't possibly distinguish itself on the "high performance" front. As for "lightweight".... you can use ZMQ for intra-process communication. That's lightweight - not using an external RDBMS.
I'm not saying this project is bad - I have no way of knowing. It might have legitimate usecases where it excels. But what it advertises can't be true.
Re: A lightweight, high-performance, language-independent job queue system
#20How is the performance, how good does it scale horizontal/vertical?
Could it be used as a replacement to Kafka message queue/ring?
(me is still looking for a Kafka-like piece coded in Go)