First version of the Java client available here: https://github.com/xetorthio/jedisque
Disque – a distributed message broker
91–96 of 96 posts
Re: Disque – a distributed message broker
#92I've created a Go client for Disque, available here:
https://github.com/zencoder/disque-go
Re: Disque – a distributed message broker
#93Earlier quoted context omitted.
Did you come back to the RabbitMQ team with this feedback?
I did not. I probably should have, but there are only so many hours in the day, and I didn't have one for this particular task. Since I still use RabbitMQ in some places I expect I will be dealing with this more, and if the situation doesn't improve I'll contact them then.
Great to hear that. I'm part of the core team BTW and we are open to feedback.
Re: Disque – a distributed message broker
#94How do people solve the resource allocation problem with distributed job queues? By resource allocation problem, I mean that jobs may be small (so that lots of them can occur in parallel) or large (occupying a significant fraction of a machine's CPU, memory, bandwidth, whatever), and may be mixed together. Trying to do too much can effectively crash a system with OOM killer or paging. Does everybody just roll their o…
In the case of RabbitMQ, it starts paging messages to disk, IIRC. If you run into descriptor / memory / disk limits, it starts turning away new messages until it returns back to a safe threshold. There's a certain amount it can do, but ultimately, you have to apply backpressure at some point unless you're willing to start losing messages at the message broker layer. After that, there's still a lot you could do: spool…
I'm not worried about too many messages piling up; I'm worried about the consumers pulling too many messages off and acting on them concurrently using up all the resources on the machine, or pulling off too few messages, and leaving too many resources idle.
Re: Disque – a distributed message broker
#95Earlier quoted context omitted.
Out of curiosity, have you tried beanstalkd? What did you not like about it? Disque is very similar to beanstalkd, but with distribution built in.
I haven't tried it actually. I evaluated it a few years ago and I remember that the queue persistance was missing (though now I see it is supported), and it lacked handling retries out of the box (which Gearman does). Now I've wrote my own wrapper to handle retries on the client side so I might give it a try. Edit: Grammar
huh that was some years back, it has automatic retry and mature protocol.
Re: Disque – a distributed message broker
#96Can wait for the PHP client so I can finally drop Gearman
Out of curiosity, have you tried beanstalkd? What did you not like about it? Disque is very similar to beanstalkd, but with distribution built in.
We use in live environment for about 3 years. We didn't had any incident with it. Persistence is a file, that you can backup regularly to a secondary storage.
- We have on peek 10k OPs/second.
- features we like is priority, TTR, pause tube (while deploying new code)
- responds really fast to stats (we have on a Stats/Graphite dashboard)
We love the Beanstalkd Admin Console panel. https://github.com/ptrofimov/beanstalk_console