Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disque – a distributed message broker

github.com

Post reply on HN
Print view View original

51–60 of 96 posts

Re: Disque – a distributed message broker

#51

Post by cdelsolar »

pyre wrote: post #40
antirez wrote: post #22

Earlier quoted context omitted.

Let's say you have a web app that sends emails, and you have many worker processes actually sending emails. If you use something like Disque, instead of sending the email directly from the web app, you send a message to Disque, into the send_email queue: "Please send an email to foo@example.com". Workers sending emails will fetch from the send_email queue, and will get the message. Workers sending emails are supposed…

Other examples: - Queueing media files to be encoded/transcoded. - Trigger a build of your application from a web interface. Basically anything that you want to do where you might not want the web server to block until the entire task is done.

Another one is making any sort of external web request; you should never use your web server to do this.

Re: Disque – a distributed message broker

#53

Post by jpgvm »

heynk wrote: post #38
veesahni wrote: post #27

Earlier quoted context omitted.

Would love to see multi-threaded workers. That's one of the pros of using sidekiq + redis today (it uses celluloid under the hood).

I'd be interested in what it might take to swap out the redis code for disque in Sidekiq.

Not fun. Sidekiq is pretty closely married to Redis datastructures unfortunately.

However there are clones like Shoryuken [1] that would be easier to port.

[1] https://github.com/phstc/shoryuken

Re: Disque – a distributed message broker

#54

Post by DivineTraube »

While this looks promising, why not make Disque part of Redis by introducing the "message queue" data structure? This would allow to build more powerful, application-specific messaging abstractions, for instance a compare-and-swap-key-with-reliable-broadcast (CASKWREB in Redis terminology).

Re: Disque – a distributed message broker

#56

Post by nasalgoat »

Okay, looking at this I can see it as a decent replacement for Beanstalk, which has been my go-to simple queue server for years now, but has been on inactive development for some time.

I built a failover system for it involving NFS mounts and heartbeat, but having it all be automatic would be quite nice. Looking forward to a prod-ready version.

Re: Disque – a distributed message broker

#57

Post by barrkel »

How do people solve the resource allocation problem with distributed job queues?

By resource allocation problem, I mean that jobs may be small (so that lots of them can occur in parallel) or large (occupying a significant fraction of a machine's CPU, memory, bandwidth, whatever), and may be mixed together. Trying to do too much can effectively crash a system with OOM killer or paging.

Does everybody just roll their own resource-based scheduler?

Re: Disque – a distributed message broker

#58

Post by pyre »

cdelsolar wrote: post #51
pyre wrote: post #40

Earlier quoted context omitted.

Other examples: - Queueing media files to be encoded/transcoded. - Trigger a build of your application from a web interface. Basically anything that you want to do where you might not want the web server to block until the entire task is done.

Another one is making any sort of external web request; you should never use your web server to do this.

It depends on the use-case. You may want the request to your web server to wait for a response from your transaction processor (e.g. Stripe) before returning.
Post reply on HN