Hello, thanks for the comment. I hope the following notes may help to make clear what are the ideas behind the design decisions used in Disque:
> Acking is a bad idea. It requires the broker to manage a lot of state, including locking and timeouts.
I think it’s much better to put the complexity on the broker than putting it on the client. It’s definitely a tradeoff so many other people may not agree. Because of ACKs, a message will be delivered again to some client forever until there is a clear proof that the client processed the message (the ACK). The client has very little room for errors. In the Kafka model, which makes sense when you want to do stream processing, the client has to handle the storage of the offset. When the offset is committed to the broker itself, then the broker becomes a store that must guarantee certain kinds of consistency, which is fairly more complex than taking state in an AP system. Given that Disque targets workloads which are not stream oriented, where the order of messages is not very important, if not for the best-effort of trying to serve first who arrived first on the average case, to put the complexity of handling the state to the client (by managing the offset), or to the broker by solving a storage and consistency problem, seems a bad idea to me.
> Re-queuing invalidates total ordering.
This is a fundamental assumption of Disque: “causal/total order of messages is overrated”. There are a few use cases where it is a very important feature to have, there are a lot of use cases where it is not. When you don’t have to guarantee order, a lot of scalability and robustness can be added to the system since it can be AP, can handle the re-delivery of messages avoiding to put complexity to the client, can survive larger network issues, does not need any strongly consistent component which can make operations harder, and so forth.
> On the performance side, parallel distributed queue consumption (which also breaks total ordering) is directly at odds with this model.
I don’t agree with this statement. Because of this data model, different producers and consumers for the same huge queue, can split the load among many nodes in a completely scalable way. But this does not prevent scaling many unrelated queues as well. Different nodes will handle different queues, and Disque has mechanisms in order to improve affinity so that producers and consumers for the same queues will try to stay to the same nodes whenever possible.
Similarly not having to solve a storage problem (to commit clients offsets) makes a system easier to scale. To store something consistently is almost always going to cost some performance.
> Queues as opaque objects — you can only inspect by popping the top message, and you cannot access older, dequeued messages. Fortunately, Disque allows you to read the entire queue without mutating it, but it doesn't look like you can read old messages.
Disque has iterators in order to solve this problem. You can iterate, with filters, the whole messages space. However note that in the Kafka model you have the problem that the queue is a linar accessible object but without clues about what was processed and not, since the broker itself (AFAIK) has no clue about what the state of a job is: to be processed, already processed, processed 10 times with errors, and so forth. Again, this is a tradeoff, I’m much more happy with more state on the broker than on the client for the use cases Disque is designed for.
* Complicated queue topologies (fanouts, dead letter queues, etc.) become a logical necessity of the strict FIFO structure. (These topologies need to be declared every time the client starts up, and introduces the possibility of schema conflicts.)
Not sure about this since as said strict FIFO is not targeted nor possible with Disque.
* Logical de-duping is probably not possible.
This is true (or not) with all the messaging systems, basically. Storing the offset in the client side (or committing it to the broker) is the same: you need to have a consistent store, somewhere, in order to make sure after crash recovery events or network partitions you are not going to process the same messages again. Actually in order to guarantee single processing you have to store the offset as part of the transaction that produces the effects of processing the message. The same can be achieved with any messaging system that guarantees the delivery of the message, if you have a CP store somewhere. The trivial case is that you use unique IDs in order to make processing idempotent, or you use to store states in the CP store in order to make out of order delivery of messages not an issue since you ignore every message which does not match the current state. This is more the matter of moving things in some place or the other. For all the cases where single processing is desirable but not extremely important, Disque “cheap” best-effort single processing can be enough and is very scalable.
> Last point: The fact that Disque calls its messages "jobs" makes me a little disappointed that it is, in fact, not a job management system. I'd love a solid, distributed job manager.
Disque is a general messaging system, but certain design characteristics are biased towards the idea that messages will be jobs. However it is opinionated about what should be and what should not be inside the broker itself. If you want more, it’s up to the client library to implement a more full fledged job processing system on top of what Disque provides.
Last note: given that I wrote Disque to provide something to the OSS world, I was not very interested in competing with something in particular: Kafka or RabbitMQ or some other message queue. Actually the less overlap the better, so my attempt was to create something different. There are things at which existing technologies will be better and maybe there will be things at which Disque will be better.