Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RabbitMQ 4.0

github.com

Post reply on HN
Print view View original

51–60 of 120 posts

Re: RabbitMQ 4.0

#51

Post by abe94 »

brightball wrote: post #34
fidotron wrote: post #3

I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.

Is there a big advantage over Redis? Just coming from the Ruby world I know Sidekiq is extremely popular as a Redis backed queue. I know there's a RabbitMQ backed queue called Sneakers that's gained a lot of popularity as well though. Just wondering what the biggest selling points are for somebody making a decision?

We switched from a redis backed queue (bullmq) to rabbit at our company (https://github.com/laudspeaker/laudspeaker) when we hit speed issues . We found bull had a much lower throughput, after a lot of tuning compared even to rabbit out of the box.

We needed features like job priorities, and complex routing logic and on top of that our software needs to send millions of messages a minute. What we miss is a nice UI to see the state of jobs, and monitor for problems more easily like: https://taskforce.sh/

Re: RabbitMQ 4.0

#52

Post by cloverich »

brightball wrote: post #34
fidotron wrote: post #3

I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.

Is there a big advantage over Redis? Just coming from the Ruby world I know Sidekiq is extremely popular as a Redis backed queue. I know there's a RabbitMQ backed queue called Sneakers that's gained a lot of popularity as well though. Just wondering what the biggest selling points are for somebody making a decision?

Been a while since I last used, but IIRC Rabbit is much more featureful than Redis, and has built-in much of what you'd get from Redis + Sidekiq. Their concepts[1] docs provides a good overview. This may be too broad a stroke but what I liked best about it was we used it for our centralized pub sub system, and although we had services in multiple languages / frameworks, they connected to each other through Rabbit, which saved us from having to setup a bunch of different / incompatible job processing systems. Like Redis its battle tested so you know what you are getting complexity / scale wise. At my current gig we use the typical Rails / Sidekiq setup and though it works fine I definitely find myself missing Rabbit. (But in my head Redis / Rabbit have only some overlap, and seeing both at the same company would seem totally normal)

[1]: https://www.rabbitmq.com/tutorials/amqp-concepts

Re: RabbitMQ 4.0

#53

Post by hk1337 »

fidotron wrote: post #3

I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.

I know this is said a lot about things people don't like or think doesn't scale but I think I a lot of people don't set it up and use properly and it doesn't scale doing their half-baked implementation.

Re: RabbitMQ 4.0

#54

Post by tejinderss »

brightball wrote: post #34
fidotron wrote: post #3

I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.

Is there a big advantage over Redis? Just coming from the Ruby world I know Sidekiq is extremely popular as a Redis backed queue. I know there's a RabbitMQ backed queue called Sneakers that's gained a lot of popularity as well though. Just wondering what the biggest selling points are for somebody making a decision?

At my current place, using Redis with celery is becoming bottleneck for number of concurrent connections it can hold. We are using 1000 workers and start seeing issues (ceiling is 10k connections in Redis); apparently Celery creates huge number of connections. We are considering moving to RabbitMQ for same reason.

Re: RabbitMQ 4.0

#55

Post by matrix2003 »

fidotron wrote: post #3

I've regarded RabbitMQ as a secret weapon in plain sight for years. The killer reason people don't use it more is it "doesn't scale" to truly enormous sizes, but for anyone with less than a million users it's great. Too many people end up with their own half rolled pubsub via things like grpc, and they'd be far better off using this, particularly in the early stages of development.

If you like RabbitMQ, check out NATS!

I can’t speak to the new version, but it comes with support for even more messaging patterns out of the box.

Re: RabbitMQ 4.0

#56

Post by phishhook »

jihadjihad wrote: post #38
383toast wrote: post #32

does celery work with this out of the box?

It should--RabbitMQ is the default broker https://docs.celeryq.dev/en/stable/getting-started/backends-...

There are some limitations when using Quorum Queues, this was only patched recently - https://github.com/celery/celery/issues/9149

Re: RabbitMQ 4.0

#60

Post by zo1 »

tejinderss wrote: post #54
brightball wrote: post #34

Earlier quoted context omitted.

Is there a big advantage over Redis? Just coming from the Ruby world I know Sidekiq is extremely popular as a Redis backed queue. I know there's a RabbitMQ backed queue called Sneakers that's gained a lot of popularity as well though. Just wondering what the biggest selling points are for somebody making a decision?

At my current place, using Redis with celery is becoming bottleneck for number of concurrent connections it can hold. We are using 1000 workers and start seeing issues (ceiling is 10k connections in Redis); apparently Celery creates huge number of connections. We are considering moving to RabbitMQ for same reason.

Celery is an over engineered pile.. Rather move lower down the stack, with your first step being Kombu which powers celery under the hood. It's oddly "configurable" so if you need to optimize and adjust connections this is where you should go, and it's pretty interchangeable between Redis and AMQP. Really, almost a drop in replacement that should just be a config change.
Post reply on HN