Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An introduction to RabbitMQ

erlang-solutions.com

Post reply on HN
Print view View original

91–100 of 263 posts

Re: An introduction to RabbitMQ

#91

Post by nailer »

carterklein13 wrote: post #34
nailer wrote: post #8

Earlier quoted context omitted.

+1. Discovered RabbitMQ/AMQP around 2010, since then tech went through a 2015-era wave of HTTP microservices that has come, and, largely gone or moved to MQ.

When you say "gone or moved to MQ" - if not moved to messaging services like RabbitMQ/NATS/etc, where else could things have gone? At least from my experience, HTTP microservices are still very common, especially when using things like AWS Lambdas. I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc, or more and more I see lightweight systems going completely serverless and just us…

> When you say "gone or moved to MQ" - if not moved to messaging services like RabbitMQ/NATS/etc, where else could things have gone?

They could have stayed trying to do continually running microservices on HTTP.

> I feel like most continually-running backends will make use of RabbitMQ/NATS/ZeroMQ/etc

I do too.

> more and more I see lightweight systems going completely serverless and just using lambdas - which are HTTP microservices.

Likewise.

But long running HTTP microservices are lame, and everybody realises that now, despite it being a cool idea back in 2015.

Re: An introduction to RabbitMQ

#92

Post by dvlsg »

addisonj wrote: post #62
halfmatthalfcat wrote: post #13

I've been trying to rationalize using either RabbitMQ or Kafka for something I'm building. High messages per second but with more complex routing topologies. Rabbit seems to be the right path but I'm worried about scaling out as many sources seem to point as Kafka being more scalable (at least horizontally). I've been looking into Rabbit's Federation but it's still not clear if that will solve the problem down the ro…

As someone who has ran a number of messaging systems in production, this is what my current take is in general: If you are moving to a more "event-sourced" architecture, usually two main concerns (beyond basic operational stuff of uptime, scale, etc) are routing and long-term retention. RabbitMQ has the routing but not the retention. Kafka can have the retention and the routing, but it can be complex/expensive. Apach…

Would you mind expanding on some of the operational complexity you ran into with pulsar?

I think pulsar is wonderful, but I haven't had the chance to use it for anything serious / in production yet, so I'm curious what pain points you had.

Re: An introduction to RabbitMQ

#93

Post by eplanit »

carapace wrote: post #57

(Light gray thin sans-serif body font means you hate my eyes. Y U hate my eyes?)

One of many trends that needs to pass (Ooooo Apple did it, so it must be cool. I'll make mine even lighter gray, so I'm that much cooler). :-)

Reader Mode is the answer. The creator of that deserves a Nobel Prize.

Re: An introduction to RabbitMQ

#94

Post by pc86 »

halfmatthalfcat wrote: post #28
pc86 wrote: post #25

Earlier quoted context omitted.

Are you replacing an existing system that's already at scale?

No, greenfield

Then the odds of you hitting the scale where RabbitMQ v. Kafka is relevant are a million to one. There is a lot of overhead with Kafka compared to RabbitMQ.

Unless you already have Kafka infrastructure, setting up Kafka for a brand new project is crazy unless your only goal is learning how to set up Kafka.

Re: An introduction to RabbitMQ

#95

Post by fuball63 »

G4BB3R wrote: post #54

I read but I still can't understand. I would like to know a very simple example of something that can't be solved with a CRUD, and can be solved with RabbitQM

One common way queues can be used to give an async like feel to your applications and flatten out spikes of activity without having to add hardware.

So, for example, you would have a CRUD that takes requests, and when there is background work to be done, places a message on the queue, and immediately returns to the user. This frees up the server for more requests. Meanwhile in the background, a worker process chugs through the queue and does its work. During long spikes it will take longer to get through the queue, but your end users will not have disruption of service.

Re: An introduction to RabbitMQ

#98

Post by 0xkd »

danielstocks wrote: post #37

I used RabbitMQ together with python and celery quite extensively and it scales really well. One thing we had trouble with though was to find a nice mechanism to scheduled tasks. Eg. “Run this task 12 hours before departure”. Maybe AMQP is the wrong place to solve that problem.

Celery has eta/countdown params that allow for running tasks at a specific time

Re: An introduction to RabbitMQ

#99

Post by heipei »

dkersten wrote: post #39
akyu wrote: post #5

RabbitMQ is great. One of the few pieces of software I've used that "just works". The only downside is once you get message-queue-pilled, you start seeing opportunities to refactor/redesign with message queues everywhere and it can be hard to resist the urge. It really is remarkable how, when used appropriately, message queues can dramatically simplify a system.

How is it for production deployment? I was considering it for something recently, but got overwhelmed by the documentation on setting up a fault-tolerant production deployment, so have been avoiding it. Was this an overreaction? What is your experience with that? Also, do you happen to know how well it works in a fault-tolerant way for communicating between services that are in different data centers? My main use-cas…

My go-to solution for fault-tolerant message queues is nsq (https://nsq.io/). nsq works differently from most other message queues in that it's supposed to be run in a distributed fashion, i.e. one nsqd running wherever messages are produced. That way you have a lightweight and fast local message queue that you can push messages to and not worry about network connectivity. You can use nsqlookupd to find the distributed nsqd that hold the topic you want to subscribe to, or you can run an additional nsq-to-nsq process to push messages from one broker to the next. It's a really great and very mature and stable piece of software. I'd say the only downside to using nsq is that you have to invest a little more in monitoring and you have to make sure that network connectivity between your consumer and each nsqd that carries a certain topic is possible.

Re: An introduction to RabbitMQ

#100

Post by gog »

dkersten wrote: post #72
zonk_ wrote: post #50

Earlier quoted context omitted.

We use it in a fairly big scale for our slack bot system. It was just set up once, as akyu said, and since then it just works. Whenever we had troubles, it was always anything other than RabbitMQ. I've also looked into other solutions (ActiveMQ, Google PubSub, ...) and RabbitMQ is by far the most straight-forward and quick to set up. There are some edge cases that it doesn't cover as well, for example automatic retri…

Was it easy to setup in terms of reliability and failover? Given what you and larrik are saying, I think I need to give it a trial run, but its a project with a tiny team, so I want to be sure it won't be the cause of sleepless nights when things go wrong. It sounds like RabbitMQ is quite solid and shouldn't be the cause for concern, which is promising! Is there anything I should keep in mind for running it in produc…

I've been running RabbitMQ for >8 years in production, once even in a fleet of 180 buses where every bus had an instance of rabbitmq running locally.

Never had a single issue in all those years.

But, I must admit that running a HA cluster is something that I've never tried, it sounds complicated and scary once you start digging through the docs.

All my deployments have been to bare metal Ubuntu and Debian machines with durable Qs and messages.

If you need to use transactions, they are really slow, couple of orders of magnitude slower compared to regular AMQP usage.

Post reply on HN