Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How to use RabbitMQ in service integration

erlang-solutions.com

Post reply on HN
Print view View original

31–40 of 44 posts

Re: How to use RabbitMQ in service integration

#31

Post by jpalomaki »

pyepye wrote: post #4
nesarkvechnep wrote: post #3

When I proposed to use RabbitMQ for service integration my colleagues didn't agree to go with it because it "won't scale" and AWS doesn't provide a managed service. Because of an AWS fetish and awful development experience the product won't launch anytime soon.

AWS released the option to have a managed RabbitMQ broker under Amazon MQ back in November (2020) https://aws.amazon.com/about-aws/whats-new/2020/11/announcin...

One thing to note is the pricing. Clustering is not supported with the micro instance, so if you need it, you must go with at least mq.m5.large [1] and you need three of them, so the total cost is around $620 per month (or closer to $700 for example in eu-central-1).

[1] https://docs.aws.amazon.com/amazon-mq/latest/developer-guide...

Re: How to use RabbitMQ in service integration

#32

Post by [deleted] »

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

[deleted]

Re: How to use RabbitMQ in service integration

#33

Post by varispeed »

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

I know workplaces where you can get sacked just for mentioning RabbitMQ...

Re: How to use RabbitMQ in service integration

#34

Post by datavirtue »

zomglings wrote: post #6
cratermoon wrote: post #5

Earlier quoted context omitted.

At a previous employer I had what I thought was close to a slam dunk case for messaging with RabbitMQ. While I'm not the most experienced person in developing distributed messaging systems, and I expected technical objections, the disheartening response, and ultimate reason for rejection, was simply organizational inertia. In short, "you can't use RabbitMQ and messaging for that because we don't use messaging here".…

I can sympathize with your colleagues. Developing the distributed messaging system is not where the bulk of the cost of running such a system goes - it's the maintenance and debugging that soak up time, money, and tears. While my past experiences with RabbitMQ in production have been stellar, I can see why a team would be hesitant to add this complexity to their infrastructure.

All of these problems sound like they come from developers who don't know how to develop distributed systems. If anything this should be simplifying an otherwise intractible problem. I have seen people overuse messaging and had to fix what I could. I otherwise reach for RMQ as a simple solution to horizontal scaling and I write the software to easily switch brokers (Aws SQS, on prem RMQ, cloud RMQ). It's not something that you choose to adopt and then force developers to make every interaction a distributed message.

Re: How to use RabbitMQ in service integration

#35

Post by manigandham »

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

Pulsar is great but also a heavy install. There are lots of messaging products now and I would recommend NATS as the best 1:1 replacement of RabbitMQ if you need a message broker with advanced routing: https://nats.io/

Re: How to use RabbitMQ in service integration

#36

Post by manigandham »

Serow225 wrote: post #15
jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

I feel dumb saying it, but as someone whose had a lot of experience with messages buses (Rabbit and AMQP1.0), I've always struggled to understand what domains/situations Kafka is actually the best fit for. It's probably because of the areas that I work in which doesn't make it obvious, but I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :) RabbitMQ is on…

Kafka was designed for scale and developed at LinkedIn to meet their extremely high throughput. It's a distributed log, basically writing to an append-only file that's partitioned by a hash of a key (that can be set on every message).

It makes the brokers as dumb as possible to optimize for performance and the logic sits in the client. You can ask to read back from the log at any point or at the current tail. Acknowledging messages is just writing a bookmark to another topic saying where you last read up to, or you can keep track of it yourself somewhere else.

You can always build more complex logic on top which Confluent has done with things like ksqldb.

Re: How to use RabbitMQ in service integration

#37

Post by winrid »

pkb wrote: post #14

DON'T. RabbitMQ is fragile and breaks under low memory/high swap conditions. Usually due to epmd breaking. I know, have to fix it (usually by killing epmd) daily.

I've seen downstream services fail and RMQ have to swap hundreds of millions of messages to disk, and then resume by itself as consumers picked up... maybe this issue you see affects certain versions?

Re: How to use RabbitMQ in service integration

#38

Post by turtlequeue »

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

While RabbitMQ is not a “never again” for me, I agree that making it work reliably does involve a few knobs and architectural tricks. I have been using AMQP/Rabbit/Kafka previously (in that order), and switched to Pulsar where I can since 2017 ( https://stackoverflow.com/a/47477765 ).

It has been great overall.

So good I have recently decided to slow down client work and build a managed SaaS offering for Pulsar: https://turtlequeue.com It is a work in progress, however it is a bit different from the nascent Pulsar offerings out there.

The main goal are ease of use and being cheap. How do I go about it?

1. Behind the scenes there is only one pulsar cluster. This lowers the costs of hosting dramatically. Even the smallest production pulsar cluster requires:

  - ZooKeeper node(s)
  - Bookies nodes
  - Brokers node(s)
  - (optional) Function workers node(s)
  - (optional) Proxies node(s)
  - Pulsar Manager
  - Prometheus
  - Grafana
.. typically this runs on top of kubernetes these days, so throw in volume storage and a LoadBalancer. Hosting small setups is costly. By having a shared cluster I can lower the costs enough to provide a free “try me” service at little to no cost to me. And nobody will suffer from the “noisy neighbour” as Pulsar is designed to be multi-tenant and can enforce limits per tenant.

2. Tq (turtlequeue) users do not have to care about how the cluster operates (typical SaaS). It is also dramatically easier for me to monitor and operate only one cluster.

3. How do I expose this safely and make it easy for users to use Pulsar then? Experienced Pulsar users will notice that this is not easy to do at the moment with pulsar. I am developing a custom proxy! This in turns allows me to collect metrics/enforce finer permissions, present a nicer dashboard.

Where am I now? The custom proxy works, the website/docs/login/dashboard/metrics/pricing need a lot of TLC. So “soon”. I will be looking for beta testers, if you are interested please email turtle@turtlequeue.com Feel free to email me too if you just want to be kept in the loop :)

Re: How to use RabbitMQ in service integration

#39

Post by mianos »

Same here in regards to liking RabbitMQ until you need to scale it. I wasted a huge amount of time years ago trying to build a HA cluster that worked when you hit it with a load.

What didn't help was the pika python client had so many issues. One of my github problem reports had a 20 line demo showing how it broke with anything but a trivial load. I gave up soon after as I had further problems in other languages. Over a year later someone looked at it and said 'yep fails under load' and fixed it.

One of those projects that showed so much promise only to make me sad.

Re: How to use RabbitMQ in service integration

#40

Post by jpgvm »

Serow225 wrote: post #15
jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

I feel dumb saying it, but as someone whose had a lot of experience with messages buses (Rabbit and AMQP1.0), I've always struggled to understand what domains/situations Kafka is actually the best fit for. It's probably because of the areas that I work in which doesn't make it obvious, but I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :) RabbitMQ is on…

Kafka is distinctly different from enterprise messaging systems like AMQP.

I generally think of messaging systems falling into 4 distinct categores: PubSub, Streaming, Queues and Enterprise Messaging Systems.

PubSub sytems are focused on non-durable (usually), low latency messaging generally without acknowledgements and generally at-most-once. i.e things like Redis PUBSUB, NATS, etc

Queues are generally focused on fanout to multiple consumers with at-least-once processing of durable messages with acknowledgements. i.e Celery/Sidekiq, Que, AWS SQS.

Streaming systems are designed for throughput and usually are based on some form of a distributed log concept. Generally offload offset management to consumers. i.e Kafka, Kinesis

Enterpise Messaging Systems favor flexibility above all else and usually have some mechanism of encoding the flow of data separately from the applications themselves. i.e exchange routing topologies in AMQP as an example. They can generally implement pubsub, queues and direct messaging paradigms. Tradeoffs being poorer availability, complexity and poor performance vs specialised systems. i.e RabbitMQ, HornetMQ, etc.

So you end up using Kafka when it's limitations aren't a problem and you need the throughput. It works best when each and every message in a stream is homogenous as such failure to process a message is unlikely to be independent of failure to process a following message. This alleviates the main drawback of streaming systems which is head of line blocking.

Some cases where it works very well is event streams, data replication/CDC, etc.

Post reply on HN