Disque 1.0 RC1 is out

However, if some constructive criticism is permitted, I have to say that, having written distributed applications for many years, I have come to dislike the "classical" push/pop queue data model:

* Acking is a bad idea. It requires the broker to manage a lot of state, including locking and timeouts.

* Re-queuing invalidates total ordering.

* On the performance side, parallel distributed queue consumption (which also breaks total ordering) is directly at odds with this model.

* Queues as opaque objects — you can only inspect by popping the top message, and you cannot access older, dequeued messages. Fortunately, Disque allows you to read the entire queue without mutating it, but it doesn't look like you can read old messages.

* Complicated queue topologies (fanouts, dead letter queues, etc.) become a logical necessity of the strict FIFO structure. (These topologies need to be declared every time the client starts up, and introduces the possibility of schema conflicts.)

* Logical de-duping is probably not possible.

Apache Kafka gets the data model right. It wisely acknowledges that queues are linear and should stay that way: In Kafka, queues are strictly append-only logs where every consumer has a cursor to the last position it read. In this model, many of the classical concerns melt away: Acks/nacks are unnecessary (consumers simply "commit" their position); total ordering is always preserved (since the queue cannot be reordered) and parallelism is made explicit (through named partitions); de-duping is trivial, and complicated topologies are largely eliminated (AMQP-type "exchanges" that fan out to separate queues are unnecessary because multiple readers can all consume the same queue without changing it, as their position is independent of the queue); and you get to choose either at-most-once or at-least-once delivery consistency by how carefully you manage your offset.

Since logs are strictly linear, you are also given the choice of how much history to keep — all of it, if you want — which opens up some interesting use cases that are not possible with classical brokers.

Kafka isn't perfect. It's a huge pain if you're not in Java land. There are no modern, mature "high-level" client implementations for Go, Ruby or Node.js. Its reliance on the JVM and on ZooKeeper makes it fairly heavyweight, both on the server and on the consumer side (the new API for broker-stored offsets simplifies things, but non-Java clients are far behind). I would really love to have a lighter-weight, language-agnostic Kafka-like implementation without the Java baggage.

Last point: The fact that Disque calls its messages "jobs" makes me a little disappointed that it is, in fact, not a job management system. I'd love a solid, distributed job manager.