good question in the blog's comments:
How is this compared to 0mq or nanomsg?
Disque 1.0 RC1 is out
31–40 of 54 posts
Re: Disque 1.0 RC1 is out
#32Looking forward to Aphyr's Call Me Maybe for Disque.
This was my first thought also. aphyr seems to be the go to for finding flaws in everyone's distributed systems.
Re: Disque 1.0 RC1 is out
#33Looking forward to Aphyr's Call Me Maybe for Disque.
Someone else could also do the analysis, Jepsen is open source
Re: Disque 1.0 RC1 is out
#34Looking forward to Aphyr's Call Me Maybe for Disque.
This was my first thought also. aphyr seems to be the go to for finding flaws in everyone's distributed systems.
If you look at the last paragraph, it seems aphyr has been involved in the development:
> I was not alone during the past months, while hacking with Disque and trying to figure out how to shape it, I received the help of: [...] Kyle Kingsbury, [...] Redis Labs and Pivotal, and probably more people I’m not remembering right now. Thank you for your help.
Re: Disque 1.0 RC1 is out
#35How does this (Disque) compare to Kafka?
What are the differences and same?
Re: Disque 1.0 RC1 is out
#36Disque is definitely exciting, and looks like it can replace RabbitMQ, which has serious flaws in its clustering design. I'm looking forward to trying it out. However, if some constructive criticism is permitted, I have to say that, having written distributed applications for many years, I have come to dislike the "classical" push/pop queue data model: * Acking is a bad idea. It requires the broker to manage a lot of…
Interesting comment!
Would love to read antirez response to these points. Oversights, omissions, alternative design?
Re: Disque 1.0 RC1 is out
#37Earlier quoted context omitted.
Could you please elaborate on these flaws within RabbitMQ's clustering design? I'm legitimately interested in a discussion on the matter.
It's extremely dangerous to run in production because it will lose data [1] by design. RabbitMQ does not have a good strategy for recovering from partitions, which happens when a node is unable to talk to its peers. Partitions can occur not just from actual network hiccups but also simply due to high CPU or I/O load or benign VM migrations. The underlying cause is that RabbitMQ is not multi-master by default. A queue…
I wouldn't call this a design flaw, it just shifts the responsibility of handling conflicts to the consumer app rather than forcing you into a server-side solution. It's too application-specific what kind of partition recovery do you need. Most people don't mind to lose a message. If you don't want to lose messages (CP-mode?) just stick all clients for certain app object (AMQP tree) to consume from the same node and block until everything is healthy or fail them over in some app-specific way to ensure consistency. While other AMQP servers may handle it better for your case these things tend to be app-specifc since your app knows better than the server which trees are critical and which are not.
Re: Disque 1.0 RC1 is out
#38Earlier quoted context omitted.
It's extremely dangerous to run in production because it will lose data [1] by design. RabbitMQ does not have a good strategy for recovering from partitions, which happens when a node is unable to talk to its peers. Partitions can occur not just from actual network hiccups but also simply due to high CPU or I/O load or benign VM migrations. The underlying cause is that RabbitMQ is not multi-master by default. A queue…
I wouldn't call this a design flaw, it just shifts the responsibility of handling conflicts to the consumer app rather than forcing you into a server-side solution. It's too application-specific what kind of partition recovery do you need. Most people don't mind to lose a message. If you don't want to lose messages (CP-mode?) just stick all clients for certain app object (AMQP tree) to consume from the same node and…
I would definitely call it a design flaw. After all, RabbitMQ pretends to be multi-master; but if you set up clients to treat a single node (of several) as a master, the other nodes will be reduced to being dumb backup nodes.
Also, don't forget that clients don't decide which node owns a queue — it's owned by the node where it's first created. Being consistent about always talking to the correct node, and keeping track of which node owns the master replica of any given queue, puts a complicating burden on both clients and the system administrator.
You can run RabbitMQ without any partition handling, but that means the ops staff needs to wake up in the middle of the night to handle downtime. Not to mention that there's no queue merging support — at the very least, it could be possible to merge conflicting queues where you don't particularly care about ordering.
Re: Disque 1.0 RC1 is out
#39good question in the blog's comments: How is this compared to 0mq or nanomsg?
ZeroMQ is not a messaging broker, it's a socket abstraction. (I interpret the zero in ZeroMQ to mean "no".) So they are not comparable at all. You can build a broker with ZeroMQ, which people have done [1], in the same sense that you can build a broker on plain TCP sockets.
I believe the same applies to Nanomsg.
[1] https://github.com/zeromq/malamute, https://github.com/sintaxi/waterfront, https://github.com/zeromq/zbroker
Re: Disque 1.0 RC1 is out
#40> However I’m not living into the illusion that I got everything right in the first release, so it will take months (or years?) of iteration to really reach the operational simplicity I’m targeting. It's always refreshing to hear such good programmers acknowledging how hard building complex systems is.
... and this is a real professional talk. I'm tired of newbies being baselessly overconfident...