How to use RabbitMQ in service integration • Hacker News Board

Re: How to use RabbitMQ in service integration

#21

Post by rad_gruchalski » Mon, May 17, 2021, 5:16 PM UTC

Serow225 wrote: post #15

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

I feel dumb saying it, but as someone whose had a lot of experience with messages buses (Rabbit and AMQP1.0), I've always struggled to understand what domains/situations Kafka is actually the best fit for. It's probably because of the areas that I work in which doesn't make it obvious, but I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :) RabbitMQ is on…

Well, they’re two different things. AMQP 1.0 is everyone to everyone messaging where every party can be a consumer and a server. RabbitMQ, traditionally, is a queue. You add a message, you take it and lock it, if no processing confirmation, it is released back for someone else to process. Kafka is an append only log. You put a message in and consumers just roll over them. Rabbit/amqp is random access, Kafka sucks at random access. With amqp, you’ll have hundreds/thousands of queues, this may be difficult with Kafka.

You’d use Kafka more as an unbounded buffer and build different paradigms on top of it. It not unusual to ingest 100s if mbits of data into kafka, potentially saturating the network while also reading that much out. Amqp is better for large number of queues where each queue has less messages in. Think mqtt, websockets - many, many consumers.

It would be reasonable to use both next to each other.

But I’d never go for rabbitmq. I’d go for azure servicebus or artemis with qpid.

Re: How to use RabbitMQ in service integration

#22

Post by rad_gruchalski » Mon, May 17, 2021, 5:29 PM UTC

eternalban wrote: post #16

I've skimmed the comments here and of course the article. Some points based on experience with Kafka, RabbitMQ and exploratory (and ahead of hype cycle) look at Pulsar: RabbitMQ is an excellent messaging middleware. But simply remember that it is not designed for optimal performance when holding on to data. "It is a river, not a lake". Performance is very sensitive to the amount of data that is in flight between send…

Mostly agreed but I’d say: Pulsar is a lake, Kafka is a river, RabbitMQ is a creek. Kafka still moves fast and once you out of retention, difficult to read past. Pulsar, thanks to adding bookies to the storage pool, can grow and does not have a retention to commit to.

Re: How to use RabbitMQ in service integration

#23

Post by closeparen » Mon, May 17, 2021, 5:40 PM UTC

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

>For the longest time I just made do with Kafka but this had serious deficiencies when implementing queues because of the cumulative ack only nature of Kafka.

We built a Kafka consumer that's effectively capable of selective acks by producing bad messages to separate topics. It's a little silly but it works.

Re: How to use RabbitMQ in service integration

#24

Post by suresk » Mon, May 17, 2021, 5:44 PM UTC

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

Pulsar definitely looks like a great combo of capabilities from Kafka + AMQP. I have been wanting to try it out with some of our stuff, but inertia/time constraints have made it hard to consider moving away from Rabbit.

I'm curious what makes it go on your "never again" list? We've definitely had our fair share of issues with it, namely -

- Really easy to misconfigure queues/exchanges, especially trying to do something like have a retry + DLQ setup.

- If you have a queue build up to a large number of messages (100 million+) for whatever reason, purging it will probably bring down the cluster.

Overall, our experience has been mostly positive. It isn't on my "never again" list, but I'm definitely wary of some parts of it and it is on my list of one of the more difficult pieces of our infrastructure to scale.

Re: How to use RabbitMQ in service integration

#25

Post by jayd16 » Mon, May 17, 2021, 5:46 PM UTC

reacharavindh wrote: post #18

jayd16 wrote: post #13

Earlier quoted context omitted.

Hmm, what issues did you run into? I've used it on a few projects in a mirrored way and it was always fine. Is the clustering the issue?

Not OP, but that is my experience. It worked like a rock on a single server. Clustering brought us issues rooting from its complexity. Split brain scenario, corruption of the Mnesia database and such. We went back to single server mode.

>Mnesia

Is Mnesia just terrible or is there some trick? I did run into issues you're talking about with ejabberd clusters.

Re: How to use RabbitMQ in service integration

#26

Post by KptMarchewa » Mon, May 17, 2021, 6:51 PM UTC

closeparen wrote: post #23

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

>For the longest time I just made do with Kafka but this had serious deficiencies when implementing queues because of the cumulative ack only nature of Kafka. We built a Kafka consumer that's effectively capable of selective acks by producing bad messages to separate topics. It's a little silly but it works.

DLQ is the way to do this. Nothing silly about it.

Re: How to use RabbitMQ in service integration

#27

Post by closeparen » Mon, May 17, 2021, 7:08 PM UTC

KptMarchewa wrote: post #26

closeparen wrote: post #23

Earlier quoted context omitted.

>For the longest time I just made do with Kafka but this had serious deficiencies when implementing queues because of the cumulative ack only nature of Kafka. We built a Kafka consumer that's effectively capable of selective acks by producing bad messages to separate topics. It's a little silly but it works.

DLQ is the way to do this. Nothing silly about it.

It's a heavyweight custom client to hack Kafka into being something it isn't, that other message brokers are. That's what's silly.

Re: How to use RabbitMQ in service integration

#28

Post by robalfonso » Mon, May 17, 2021, 7:20 PM UTC

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

Wow, and here I was feeling like I was the only one who thought this way! Such a similar experience to yours.

Re: How to use RabbitMQ in service integration

#29

Post by datavirtue » Mon, May 17, 2021, 7:28 PM UTC

bironran wrote: post #12

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

Can only echo parent. 3 places of work in varying sizes, 4 projects in varying maturity, not a single RMQ administration staff that was competent enough to reliably run the cluster. Which of course leads me to believe the problem isn't with the people but with the ridiculously high threshold of knowledge, experience and app developer self-control needed to run RMQ successfully. As parent said, many meltdowns later, I…

I concure. I like RMQ because I know how to configure and administer it but I would never trust anyone with it. My first exposure to it was on a project where the lead architect failed to read the documentation let alone understand any of it. Several years later I was able to fix it and then left it in the hands of some other incompetents.

Re: How to use RabbitMQ in service integration

#30

Post by reader_1000 » Mon, May 17, 2021, 7:36 PM UTC

Serow225 wrote: post #15

jpgvm wrote: post #11

RabbitMQ went on my "never again" list of things after dealing with it for years as part of OpenStack and projects before that. At the time there wasn't many viable alternatives to AMQP that were OSS and reasonable. However many split brains and grey hairs later I decided RabbitMQ was almost never worth it regardless of how many of AMQPs advanced features you could make use of. For the longest time I just made do wit…

I feel dumb saying it, but as someone whose had a lot of experience with messages buses (Rabbit and AMQP1.0), I've always struggled to understand what domains/situations Kafka is actually the best fit for. It's probably because of the areas that I work in which doesn't make it obvious, but I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :) RabbitMQ is on…

> I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :)

If order of message processing matters, then Kafka is better suited then AMQP. For example, In a distributed application for money transfers, if AMQP used, message order will be lost and some problems will occur in the following scenario:

User A with an accound of $1000 makes order for two transfers T1 ($600) and T2 ($500)

  - Rabbit delivers T1 to server1, before processing message, server1 enters a full GC.
  - Rabbit delivers T2 to server2 and server2 processes message immediately, now User A's account have $500  
  - Server1 resumes its life after the end of GC, but fails to process T1 since account's balance is less than required amount.

However, it is T2 that should have failed because User A ordered T1 first and T2 after.

In Kafka, when user account identifier is used for partitioning key, all User A's messages will be processed by same consumer (i.e server1), so even if server1 enters a full GC, that is OK, since T2 will be processed after T1.