Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Disque – a distributed message broker

github.com

Post reply on HN
Print view View original

61–70 of 96 posts

Re: Disque – a distributed message broker

#61

Post by Blackthorn »

barrkel wrote: post #57

How do people solve the resource allocation problem with distributed job queues? By resource allocation problem, I mean that jobs may be small (so that lots of them can occur in parallel) or large (occupying a significant fraction of a machine's CPU, memory, bandwidth, whatever), and may be mixed together. Trying to do too much can effectively crash a system with OOM killer or paging. Does everybody just roll their o…

Kind of a different problem, isn't it?

My first exposure to the resource allocation problem and solutions came at Google, whose system evolved into Kubernetes (now an open source project). It's so damn effective I hope it takes off everywhere.

Re: Disque – a distributed message broker

#62

Post by agopaul »

orthecreedence wrote: post #60
agopaul wrote: post #31

Can wait for the PHP client so I can finally drop Gearman

Out of curiosity, have you tried beanstalkd? What did you not like about it? Disque is very similar to beanstalkd, but with distribution built in.

I haven't tried it actually. I evaluated it a few years ago and I remember that the queue persistance was missing (though now I see it is supported), and it lacked handling retries out of the box (which Gearman does).

Now I've wrote my own wrapper to handle retries on the client side so I might give it a try.

Edit: Grammar

Re: Disque – a distributed message broker

#63

Post by barrkel »

Blackthorn wrote: post #61
barrkel wrote: post #57

How do people solve the resource allocation problem with distributed job queues? By resource allocation problem, I mean that jobs may be small (so that lots of them can occur in parallel) or large (occupying a significant fraction of a machine's CPU, memory, bandwidth, whatever), and may be mixed together. Trying to do too much can effectively crash a system with OOM killer or paging. Does everybody just roll their o…

Kind of a different problem, isn't it? My first exposure to the resource allocation problem and solutions came at Google, whose system evolved into Kubernetes (now an open source project). It's so damn effective I hope it takes off everywhere.

Well, my first concern about a "job queue" (as opposed to a message queue) is getting work done efficiently and reliably; do too little, and resources aren't being used efficiently, but do too much, and you might never get anything done.

We have this exact problem at the startup where I work. We have a home-made solution (a combo of Akka and Play for API / admin UI) that works OK, but really we'd prefer not to be in the business of writing job queues and schedulers.

Something big and cluster-oriented like Hadoop isn't a good fit; we typically only have one or two actual machines servicing jobs, because of our business model. Financial entities don't like their data being mixed with other people's data, so we give everybody tiny little networks of VMs, and can't farm work out to a giant cluster.

Redis is used as the backing storage for our homemade job queue. But without resource monitoring and allocation, Disque doesn't help with our problem. I'm sure it'll find much use elsewhere, though.

Re: Disque – a distributed message broker

#64

Post by old_sound »

IgorPartola wrote: post #48
lobster_johnson wrote: post #41

Earlier quoted context omitted.

> why rabbitmqctl and rabbitmqadmin rabbitmqctl is the core tool to interact with a node. It can show the cluster status, status of the node process etc., stop and start the app and so on. It works at a lower level. rabbitmqadmin comes with the Management plugin, and is a client for the plugin's HTTP API. You have to enable the plugin to expose that API. The Management plugin adds some overhead, I believe (it samples…

Thanks for that. Yes, I figured that out at some point. Unfortunately, IIRC there were cases where I needed to accomplish what seemed like pretty basic things, but they could only be done with the Management plugin. This was on the order of "does this user's password hash match what I have?" The reason was to get Puppet to perform configuration management on the node. It worked eventually, but cost me an hour or two,…

Did you come back to the RabbitMQ team with this feedback?

Re: Disque – a distributed message broker

#65

Post by jonny_eh »

pyre wrote: post #58
cdelsolar wrote: post #51

Earlier quoted context omitted.

Another one is making any sort of external web request; you should never use your web server to do this.

It depends on the use-case. You may want the request to your web server to wait for a response from your transaction processor (e.g. Stripe) before returning.

That shouldn't be done in a web request. Not only can it lead to wasted time in the request handler, but it can lead to race conditions like double purchases.

Re: Disque – a distributed message broker

#68

Post by tannerburson »

jpgvm wrote: post #53
heynk wrote: post #38

Earlier quoted context omitted.

I'd be interested in what it might take to swap out the redis code for disque in Sidekiq.

Not fun. Sidekiq is pretty closely married to Redis datastructures unfortunately. However there are clones like Shoryuken [1] that would be easier to port. [1] https://github.com/phstc/shoryuken

For giggles I spent an hour this evening and added support for Disque to the pluggable system I help maintain [0].

Things I can say: the ruby disque library isn't really fleshed out yet. It's alpha, so that's fine, but it's got a ways to go. For example, it doesn't directly expose ACKJOB as a command. The server is equally alpha, things like HELP don't do anything yet, and some options on some commands appear broken. But hey, it was a fun way to spend an hour.

[0]https://github.com/tannerburson/chore-disque.

Re: Disque – a distributed message broker

#69

Post by brndnmtthws »

barrkel wrote: post #57

How do people solve the resource allocation problem with distributed job queues? By resource allocation problem, I mean that jobs may be small (so that lots of them can occur in parallel) or large (occupying a significant fraction of a machine's CPU, memory, bandwidth, whatever), and may be mixed together. Trying to do too much can effectively crash a system with OOM killer or paging. Does everybody just roll their o…

It may be worth reading about Mesos: https://www.cs.berkeley.edu/~alig/papers/mesos.pdf

Re: Disque – a distributed message broker

#70

Post by seabrookmx »

atombender wrote: post #43

This looks great. Are there any plans for topic-based pubsub? With AMQP (specifically, RabbitMQ) I can set up a fanout topology such that a publisher simply publishes to an exchange. Clients then bind their queues to this exchange based on a routing key, and any messages matching the key will be copied there. If no clients have bound, messages disappear into the aether. For example, the publisher can use the exchange…

> If no clients have bound, messages disappear into the aether

To everyone who thinks this sounds scary, don't worry. You can bind dedicated "Undelivered queues" and "Dead letter" queues to exchanges to make sure that when routing fails, you don't lose any messages.

We're using RabbitMQ in a few newer projects and it's really a joy to work with!

Post reply on HN