How to use RabbitMQ in service integration • Hacker News Board

Re: How to use RabbitMQ in service integration

#41

Post by cratermoon » Wed, May 19, 2021, 6:21 PM UTC

datavirtue wrote: post #34

zomglings wrote: post #6

Earlier quoted context omitted.

I can sympathize with your colleagues. Developing the distributed messaging system is not where the bulk of the cost of running such a system goes - it's the maintenance and debugging that soak up time, money, and tears. While my past experiences with RabbitMQ in production have been stellar, I can see why a team would be hesitant to add this complexity to their infrastructure.

All of these problems sound like they come from developers who don't know how to develop distributed systems. If anything this should be simplifying an otherwise intractible problem. I have seen people overuse messaging and had to fix what I could. I otherwise reach for RMQ as a simple solution to horizontal scaling and I write the software to easily switch brokers (Aws SQS, on prem RMQ, cloud RMQ). It's not somethin…

As I noted, I'm definitely not the most experience in implementing messaging system. Can you recommend some articles or books on distributed systems and messaging that I should know?

Re: How to use RabbitMQ in service integration

#42

Post by Serow225 » Mon, May 24, 2021, 6:51 PM UTC

reader_1000 wrote: post #30

Serow225 wrote: post #15

Earlier quoted context omitted.

I feel dumb saying it, but as someone whose had a lot of experience with messages buses (Rabbit and AMQP1.0), I've always struggled to understand what domains/situations Kafka is actually the best fit for. It's probably because of the areas that I work in which doesn't make it obvious, but I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :) RabbitMQ is on…

> I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :) If order of message processing matters, then Kafka is better suited then AMQP. For example, In a distributed application for money transfers, if AMQP used, message order will be lost and some problems will occur in the following scenario: User A with an accound of $1000 makes order for two transfers T1…

FWIW, AMQP1.0 does support Sessions which can be used to address this scenario :)

Re: How to use RabbitMQ in service integration

#43

Post by Serow225 » Mon, May 24, 2021, 6:55 PM UTC

rad_gruchalski wrote: post #21

Serow225 wrote: post #15

Earlier quoted context omitted.

I feel dumb saying it, but as someone whose had a lot of experience with messages buses (Rabbit and AMQP1.0), I've always struggled to understand what domains/situations Kafka is actually the best fit for. It's probably because of the areas that I work in which doesn't make it obvious, but I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :) RabbitMQ is on…

Well, they’re two different things. AMQP 1.0 is everyone to everyone messaging where every party can be a consumer and a server. RabbitMQ, traditionally, is a queue. You add a message, you take it and lock it, if no processing confirmation, it is released back for someone else to process. Kafka is an append only log. You put a message in and consumers just roll over them. Rabbit/amqp is random access, Kafka sucks at…

Thank you, that is quite helpful :)

Re: How to use RabbitMQ in service integration

#44

Post by Serow225 » Mon, May 24, 2021, 7:00 PM UTC

jpgvm wrote: post #40

Serow225 wrote: post #15

Earlier quoted context omitted.

I feel dumb saying it, but as someone whose had a lot of experience with messages buses (Rabbit and AMQP1.0), I've always struggled to understand what domains/situations Kafka is actually the best fit for. It's probably because of the areas that I work in which doesn't make it obvious, but I'd love to hear what exact scenarios it does actually make sense to use Kafka for instead of Rabbit or AMQP1.0 :) RabbitMQ is on…

Kafka is distinctly different from enterprise messaging systems like AMQP. I generally think of messaging systems falling into 4 distinct categores: PubSub, Streaming, Queues and Enterprise Messaging Systems. PubSub sytems are focused on non-durable (usually), low latency messaging generally without acknowledgements and generally at-most-once. i.e things like Redis PUBSUB, NATS, etc Queues are generally focused on fa…

Thank you, that is very helpful :) > "So you end up using Kafka when it's limitations aren't a problem and you need the throughput." I think that's what I've run into so far, is that my usecases haven't needed the throughput of Kafka and so all I'm left with is the feature gaps that I'd miss from something like AMQP1.0

Apache Pulsar looks pretty interesting :)