Live data from Hacker News

A lightweight, high-performance, language-independent job queue system

github.com

31–40 of 41 posts

Re: A lightweight, high-performance, language-independent job queue system

#31
Having the number of parallel consumers configured per-queue (as opposed to consumers dynamically being able to join and leave) seems like it imposes many of the same restrictions that make Kafka less than ideal at being a job queue.

Basically, if you have messages which can take different amounts of time to process, or you need to quickly dynamically scale the number of consumers on a queue in response to volume.

How well does the "update max_workers" queue-modification command work in situations of very high message volume and/or high consumer counts?

Re: A lightweight, high-performance, language-independent job queue system

#32
post #30

It would be interesting to hear the rationale behind using this over something like RabbitMQ, which has its own storage layer, as well as the queue aspect.

I made one of these once: https://github.com/thruflo/ntorque As other comments have pointed out, HTTP is not a great idea because timeouts, etc. The one subtle benefit that I can relay is that by using your main database as the storage layer, you can enqueue tasks within a transaction, as per: https://github.com/thruflo/ntorque/blob/master/src/ntorque/c...

Database-as-queue tends to run into performance ceilings before dedicated "true queue" systems, in my experience. As you point out, though, sticking with an RDBMS gives you nice transactionality. Additionally, using a database for a queue for as long as you can before switching also offers you the benefits of using a time-tested, usually widely-supported protocol with well documented reliability guarantees. Your queues are also easily introspectable, which can be nice (this is part and parcel with why databases tend to hit perf ceilings as queues though).

RabbitMQ also supports transactions for some queue operations, but its notion of "transaction" and what you can do inside of one is much more limited than a typical database's: https://www.rabbitmq.com/semantics.html

Re: A lightweight, high-performance, language-independent job queue system

#33
post #3

When would a push queue like this be preferable over a pull queue?

I may not be reading the documentation correctly, but I don't think this is a push queue; it uses the term "push", I think, to describe delivering messages to a queue and not as it is traditionally used to describe handing messages to consumers.

In answer to your question generally: push-based models, while more complex, tend to be higher performance (by dint of improving throughput: the broker can push messages to a consumer while it's working on other things rather than waiting for a "gimmie" request; the broker can also coordinate when and how it delivers messages to which consumers for maximum performance, which can lead to significant speedups in high throughput situations).

A very powerful pattern that gives a reasonable amount of control in a push-based situation is combining a push based model with client acknowledgements, and a client "window" of a number of messages that the client may or may not have noticed yet, especially if messages can be taken back from that window programmatically in the event of slow or dead consumers. This is what RabbitMQ/AMQP calls "Qos".

In my experience, push-based messaging models should typically not be adopted up front, unless throughput requirements are known to require such a model. The added complexity (mental and in code) of managing a push-based queue model is very rarely worth investing in at the beginning of development.

Furthermore, it's possible to have extremely high performance in a simpler pull-based model, provided you make some tradeoffs. This is what Kafka does.

I would recommend switching from pull to push only when it becomes necessary (though this can be a non-trivial amount of effort depending on how tightly integrated your code is with your messaging system). RabbitMQ/ActiveMQ/Redis/Resque as brokers will shine here, since they all support both pull models ("get" in AMQP) and push models.

Re: A lightweight, high-performance, language-independent job queue system

#34

Clearly a lot of thought and work has been put into this and I give the authors credit for that. I appreciate any entry into the space but this is a really crowded space. With some very battle tested entries in the field as people have pointed out. There are a few questions not in the readme that I think need answering for a queue system: - Execution guarantees (at least once, at most once, exactly once?) - Order gua…

Agreed, there are a lot of more robust products already in this space.

My biggest concern is its reliance on mysql. There is no way this could be a valid option for high volume messaging when it is essentially database as a queue.

Re: A lightweight, high-performance, language-independent job queue system

#35
post #4

Honest question, how is using MySQL lightweight? Many job queues I've seen use e.g. Redis

If the requirement is for persistence, i.e. no data loss if the queue process dies, then Redis won't fit. EDIT: TIL Redis has the option to turn on fsync-to-disk on every write. Probably not what people are thinking of when they suggest Redis as lightweight.

I would still call redis lightweight compared to mysql, but yes fsync always has a significant performance cost.

Re: A lightweight, high-performance, language-independent job queue system

#37

Surprised nobody has mentioned beanstalkd yet: https://kr.github.io/beanstalkd/ > Beanstalk is a simple, fast work queue. > Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously. The protocol is easy to drive directly and there are good libraries for most common languages.

Yep. Beanstalk is great for that exact use case. I've used it quite a bit and was easily able to attach different tools in different languages with no problems. PHP, Go, Python, and Java with no issues. Great tool for async workflows.

Re: A lightweight, high-performance, language-independent job queue system

#38
post #19

Earlier quoted context omitted.

Yeah, a better example would be https://github.com/zeromq/malamute

Well, my original message was a bit snarky, but what I meant was that, at the very least, this project advertises the wrong features. I mean come on, "lightweight" and "high-performance"? Maybe advertise safety properties, resilience, I don't know... but the way it is built, it can't possibly distinguish itself on the "high performance" front. As for "lightweight".... you can use ZMQ for intra-process communication.…

"Lightweight" is really becoming a turnoff for me when people use it in their description. It (mostly) means "I haven't worked on this long enough to add the features all the other projects in this space immediately found were too useful to go without" or even "I haven't worked on this long enough to produce much code".

"Lightweight" is really only interesting to me in two cases: First, you're designing it for limited resources, like an embedded system, for which the standard answer is simple too large to even consider. Second, when the standard answer in the field is so "heavy" (an ill-defined term itself, but moving on) that it causes problems of its own. JVM solutions sometimes get to this point, where the act of administering the solution itself gets bogged down in merely administering the JVM.

I do not personally have the problem that my job queues are too heavy, nor have I heard anyone else complain that ZMQ or Redis are just so heavy for what they do.

Re: A lightweight, high-performance, language-independent job queue system

#39
Hm, I've been looking for a lightweight, language-neutral job queue. RDBMS-backed is my preference since I don't need massive scalability, but do want persistence that's easy to reason about (so not Redis), transparency (so not beanstalkd) and a long-lived history (so not most of the other ones).

But, RDMBS only makes sense if you can use your existing installation, so MySQL-only is a nonstarter.

Re: A lightweight, high-performance, language-independent job queue system

#40

Surprised nobody has mentioned beanstalkd yet: https://kr.github.io/beanstalkd/ > Beanstalk is a simple, fast work queue. > Its interface is generic, but was originally designed for reducing the latency of page views in high-volume web applications by running time-consuming tasks asynchronously. The protocol is easy to drive directly and there are good libraries for most common languages.

Beanstalkd is always the queue engine I reach for. Sometimes I'll build something over SQL if it's low-volume, but I always design my queue APIs to match beanstalkd's so it can be swapped in.

Beyond being a great piece of software, I find the protocol to be really well-designed for a work queue.

Post reply on HN