Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disque – a distributed message broker

github.com

Post reply on HN
Print view View original

41–50 of 96 posts

Re: Disque – a distributed message broker

#41

Post by lobster_johnson »

IgorPartola wrote: post #33
uptown wrote: post #18

Can somebody help me better understand the possible uses for something like this?

Personally, I look forward to using this instead of Redis and RabbitMQ as the message broker for Celery. Celery is a background tasks framework for Python. You write code that defines tasks, then certain things trigger tasks. Classic example: user uploads a video, then in the background the video is re-encoded in some standard format. So after the upload is done, the web application triggers a task by sending a messa…

> why rabbitmqctl and rabbitmqadmin

rabbitmqctl is the core tool to interact with a node. It can show the cluster status, status of the node process etc., stop and start the app and so on. It works at a lower level.

rabbitmqadmin comes with the Management plugin, and is a client for the plugin's HTTP API. You have to enable the plugin to expose that API. The Management plugin adds some overhead, I believe (it samples statistics continually to serve through the API), and as a result it's optional. Not everyone would want or need to enable it.

rabbitmqctl does the basics, whereas rabbitmqadmin is a higher-level tool.

Re: Disque – a distributed message broker

#42

Post by jonny_eh »

antirez wrote: post #22
uptown wrote: post #18

Can somebody help me better understand the possible uses for something like this?

Let's say you have a web app that sends emails, and you have many worker processes actually sending emails. If you use something like Disque, instead of sending the email directly from the web app, you send a message to Disque, into the send_email queue: "Please send an email to foo@example.com". Workers sending emails will fetch from the send_email queue, and will get the message. Workers sending emails are supposed…

Isn't Redis already good for this, powering backgrounding software like Resque and Sidekiw?

What does Disque offer that's different?

BTW, thanks for Redis. It's one of my favourite pieces of software ever.

Edit: NVM, found this: http://antirez.com/news/88

Re: Disque – a distributed message broker

#43

Post by atombender »

This looks great. Are there any plans for topic-based pubsub?

With AMQP (specifically, RabbitMQ) I can set up a fanout topology such that a publisher simply publishes to an exchange. Clients then bind their queues to this exchange based on a routing key, and any messages matching the key will be copied there. If no clients have bound, messages disappear into the aether.

For example, the publisher can use the exchange "changes" to write messages with keys such as "create.post.123" or "update.user.321". A consumer can then bind a queue to the exchange with the filters "create." and "update." to get all create and update messages. A different app can listen to just ".user." to get anything related to users.

This is a really powerful tool, allowing publishers to be dumb firehoses and clients to surgically pick what they need. Of course you don't need exchanges and bindings to accomplish this, only topics.

Re: Disque – a distributed message broker

#44

Post by paulsutter »

uptown wrote: post #18

Can somebody help me better understand the possible uses for something like this?

Simple use cases for message queues in a web application include:

- Improving response latency by moving deferrable tasks out of the page serving code

- Improving response latency by simplifying implementation of concurrent tasks (whether across cores or servers)

- Allowing capacity planning for average load instead of peak load for deferrable tasks

The bigger story is that message queues make a good substrate for building fault-tolerant, highly available, scalable distributed systems that are easy to reason about and have predictable performance. This is because system components can be cleanly decoupled, implemented with redundancy and scalability, the message queue itself takes over much of the complexity of failover and delivery/ordering guarantees, and the message queue makes an excellent point for monitoring the system, giving visibility into logjams and problems in a live system.

Re: Disque – a distributed message broker

#48

Post by IgorPartola »

lobster_johnson wrote: post #41
IgorPartola wrote: post #33

Earlier quoted context omitted.

Personally, I look forward to using this instead of Redis and RabbitMQ as the message broker for Celery. Celery is a background tasks framework for Python. You write code that defines tasks, then certain things trigger tasks. Classic example: user uploads a video, then in the background the video is re-encoded in some standard format. So after the upload is done, the web application triggers a task by sending a messa…

> why rabbitmqctl and rabbitmqadmin rabbitmqctl is the core tool to interact with a node. It can show the cluster status, status of the node process etc., stop and start the app and so on. It works at a lower level. rabbitmqadmin comes with the Management plugin, and is a client for the plugin's HTTP API. You have to enable the plugin to expose that API. The Management plugin adds some overhead, I believe (it samples…

Thanks for that. Yes, I figured that out at some point. Unfortunately, IIRC there were cases where I needed to accomplish what seemed like pretty basic things, but they could only be done with the Management plugin. This was on the order of "does this user's password hash match what I have?" The reason was to get Puppet to perform configuration management on the node. It worked eventually, but cost me an hour or two, whereas complete lack of user management (and permissions managed at the network level), is more appropriate for my use cases and works just as well.

Re: Disque – a distributed message broker

#49

Post by uptown »

antirez wrote: post #22
uptown wrote: post #18

Can somebody help me better understand the possible uses for something like this?

Let's say you have a web app that sends emails, and you have many worker processes actually sending emails. If you use something like Disque, instead of sending the email directly from the web app, you send a message to Disque, into the send_email queue: "Please send an email to foo@example.com". Workers sending emails will fetch from the send_email queue, and will get the message. Workers sending emails are supposed…

Thanks very much for the explanation and the development. Fits nicely with a problem I've wanted to solve.

Re: Disque – a distributed message broker

#50

Post by IgorPartola »

misiti3780 wrote: post #35
IgorPartola wrote: post #33

Earlier quoted context omitted.

Personally, I look forward to using this instead of Redis and RabbitMQ as the message broker for Celery. Celery is a background tasks framework for Python. You write code that defines tasks, then certain things trigger tasks. Classic example: user uploads a video, then in the background the video is re-encoded in some standard format. So after the upload is done, the web application triggers a task by sending a messa…

"it's got a very simple API for examining what's going on" which api are you referring to specifically?

INFO and LLEN. By contrast, here's my command to view the length of the queue with Rabbit:

    sudo rabbitmqctl -p example list_queues name messages messages_ready messages_unacknowledged consumers | grep 'celery        \|other_celery      '
Note that I need to use sudo to get this information here. I am sure the Management plug could help here. I do have Flower set up, but it gets terribly confused about multiple queues and multiple task servers and sets of workers, so it's basically useless. It seems to latch onto one of the queues and only shows the stats for that in most of its views.
Post reply on HN