Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disque – a distributed message broker

github.com

Post reply on HN
Print view View original

81–90 of 96 posts

Re: Disque – a distributed message broker

#81

Post by antirez »

DivineTraube wrote: post #54

While this looks promising, why not make Disque part of Redis by introducing the "message queue" data structure? This would allow to build more powerful, application-specific messaging abstractions, for instance a compare-and-swap-key-with-reliable-broadcast (CASKWREB in Redis terminology).

It is not possible to replicate Disque features as a Redis data type, it is fundamentally a different best unfortunately. The project started exactly as a feature for Redis but was migrated away as a separated system because while I was working on it the tension between what Redis was and what Disque would be were bigger every day.

Re: Disque – a distributed message broker

#82

Post by juriansluiman »

mmgutz wrote: post #34

How is this different than beanstalkd?

Both are very equal it seems, with only some minor differences. Disque isn't used in production systems yet so most likely the real comparisons will arrive when Disque is more mature.

Some key differences are that beanstalkd offers (feature wise):

1. Bury / kick 2. Priorities 3. Designed for speed

And Disque:

1. Designed for HA, scalable, distributed 2. Fault tolerant 3. Peek to multiple jobs

With the amount of momentum behind Redis and the fairly unknown beanstalkd, I guess Disque will gain popularity quite soon.

Re: Disque – a distributed message broker

#84

Post by JulianMorrison »

I think the thing this is most comparable to is NSQ, but NSQ takes a different approach to distribution.

- disque: send to one, read from one. The message is handed off to N replicas, and efforts are made to avoid duplicated or dropped messages. Disque will refuse new messages if RAM is filled across the cluster.

- nsq: send to any. Read from all. NSQ nodes do not communicate or coordinate with each other. Since only one node originates the message, there's no duplication, but a node outage can drop messages, and a partition can isolate them with no consumer. NSQ can grow its queue beyond RAM, so it will keep accepting new messages even if it is partitioned from the consumer.

Personally I think NSQ's approach looks like it's doing a lot less work and achieving almost all the same guarantees.

Re: Disque – a distributed message broker

#85

Post by pyre »

cdelsolar wrote: post #76
StavrosK wrote: post #74

Earlier quoted context omitted.

Blocking is even easier, it's zero more endpoints.

Yeah and if Stripe goes down your server goes down with it.

If Stripe is your payment processor, how do you propose that one should process payments if Stripe is down? How does making payment processing asynchronous help? Why would your entire server die because Stripe's API failed to return?

Re: Disque – a distributed message broker

#86

Post by tobz »

barrkel wrote: post #57

How do people solve the resource allocation problem with distributed job queues? By resource allocation problem, I mean that jobs may be small (so that lots of them can occur in parallel) or large (occupying a significant fraction of a machine's CPU, memory, bandwidth, whatever), and may be mixed together. Trying to do too much can effectively crash a system with OOM killer or paging. Does everybody just roll their o…

In the case of RabbitMQ, it starts paging messages to disk, IIRC. If you run into descriptor / memory / disk limits, it starts turning away new messages until it returns back to a safe threshold.

There's a certain amount it can do, but ultimately, you have to apply backpressure at some point unless you're willing to start losing messages at the message broker layer.

After that, there's still a lot you could do: spool them to disk on the publisher, retry later, etc. You still risk message loss, filling up THOSE disks, etc... but again, something has to eventually give.

Re: Disque – a distributed message broker

#87

Post by tobz »

lobster_johnson wrote: post #37
jvehent wrote: post #26

Any functional advantage to using this instead of something like rabbitmq?

As a RabbitMQ user, I'll switch to the first viable alternative that is production ready. RabbitMQ's clustering isn't great. It's sensitive to partitions, which can occur not just from actual network hiccups but also simply due to high CPU or I/O load, and it does not have a good strategy to recover from such partitions. RabbitMQ is not multi-master by default. A queue is owned by a specific node, and if you have a p…

How do the AMQP frames increase in size indefinitely? (I use DLXs and message TTLs to do timed retries, without issue, although at a very low retry rate, so maybe I'm just not hitting your purported issue with increasing frame sizes?)

Re: Disque – a distributed message broker

#88

Post by IgorPartola »

old_sound wrote: post #64
IgorPartola wrote: post #48

Earlier quoted context omitted.

Thanks for that. Yes, I figured that out at some point. Unfortunately, IIRC there were cases where I needed to accomplish what seemed like pretty basic things, but they could only be done with the Management plugin. This was on the order of "does this user's password hash match what I have?" The reason was to get Puppet to perform configuration management on the node. It worked eventually, but cost me an hour or two,…

Did you come back to the RabbitMQ team with this feedback?

I did not. I probably should have, but there are only so many hours in the day, and I didn't have one for this particular task. Since I still use RabbitMQ in some places I expect I will be dealing with this more, and if the situation doesn't improve I'll contact them then.

Re: Disque – a distributed message broker

#89

Post by lobster_johnson »

tobz wrote: post #87
lobster_johnson wrote: post #37

Earlier quoted context omitted.

As a RabbitMQ user, I'll switch to the first viable alternative that is production ready. RabbitMQ's clustering isn't great. It's sensitive to partitions, which can occur not just from actual network hiccups but also simply due to high CPU or I/O load, and it does not have a good strategy to recover from such partitions. RabbitMQ is not multi-master by default. A queue is owned by a specific node, and if you have a p…

How do the AMQP frames increase in size indefinitely? (I use DLXs and message TTLs to do timed retries, without issue, although at a very low retry rate, so maybe I'm just not hitting your purported issue with increasing frame sizes?)

Every time a dead-lettering happens, RabbitMQ will do this [1]:

The dead-lettering process adds an array to the header of each dead-lettered message named x-death. This array contains an entry for each time the message was dead-lettered.

This table is never truncated. It exists in the AMQP frame and will grow indefinitely.

The AMQP spec mandates that clients negotiate a frame-max size per connection [2], and I believe RabbitMQ is in violation of the spec through this behaviour if the client specifies a non-zero value. (RabbitMQ even ignored this negotiation prior to 3.1, and there's a document [3] which indicates it ignores limits entirely.)

As a result, clients that follow frame-max strictly according to spec (such as amqplib for Node.js [4]) will refuse to handle violating frames, which is understandable given that clients also like to have predictable buffer allocation.

[1] https://www.rabbitmq.com/dlx.html

[2] https://www.rabbitmq.com/specification.html

[3] https://www.rabbitmq.com/amqp-0-9-1-errata.html

[4] https://github.com/squaremo/amqp.node

Re: Disque – a distributed message broker

#90

Post by lobster_johnson »

seabrookmx wrote: post #70
atombender wrote: post #43

This looks great. Are there any plans for topic-based pubsub? With AMQP (specifically, RabbitMQ) I can set up a fanout topology such that a publisher simply publishes to an exchange. Clients then bind their queues to this exchange based on a routing key, and any messages matching the key will be copied there. If no clients have bound, messages disappear into the aether. For example, the publisher can use the exchange…

> If no clients have bound, messages disappear into the aether To everyone who thinks this sounds scary, don't worry. You can bind dedicated "Undelivered queues" and "Dead letter" queues to exchanges to make sure that when routing fails, you don't lose any messages. We're using RabbitMQ in a few newer projects and it's really a joy to work with!

RabbitMQ is pretty good. Just be aware of its problems with clustering, as I've outlined here: https://news.ycombinator.com/item?id=9448258.
Post reply on HN