RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)

I'm sure there remain good use cases for message buses that you have to run yourself, where there really are millions of messages that can't be batched up and need real-time and whatnot. But you can get pretty far with:

1. Write a bunch of records to an S3 object. 2. Trigger a lambda to process — infinite scale out!

and if a queue really is needed due to constrained consumers, then:

1. Write a bunch of records to an S3 object. 2. The lambda trigger puts a message into an SQS queue with the S3 object's URI. 3. An auto-scaling group gets the message off the queue and processes the data in the object.

If ordering's important, then the SQS queue can be made FIFO. That has pretty low numbers in messages/second, but since records are being batched into S3 objects you can still have fairly high throughput.

It used to be that the elegance of queues for such systems would tempt developers into the operational slough of despond that is running these types of systems. Again, I'm sure it's warranted for some applications, but S3, Lambda, and SQS as above work nicely together.

Could you expand on this a bit more? I am curious.

NATs has a decent-ish guide here: https://docs.nats.io/nats-concepts/overview/compare-nats

A few things they get wrong mostly about Rabbit:

+ RabbitMQ does support replay, and also has a memory only mode which will support persistance in a cluster

+ RabbitMQ doesn't have that sensitive of a latency between cluster members (no more sensitive than NATS in some setups).

+ RabbitMQ also supports Prometheus

A good (but incomplete) rule of thumb:

+ Kafka is a distributed Append-Only-Log that looks like a message bus. It allows time travel but has very simple server semantics.

+ RabbitMQ is a true message broker with all the abilities and patterns therein. It comes with more complexity.

+ NATs is primarily a straight forward streaming messaging platform.

Also consider Redis' message queue mode, zeromq, mqtt brokers (Eclipse Mosquitto) and the option of just not using a message broker/queue system. Even as someone who really likes the pubsub pattern, there's a good chance you don't need it and you may be heading to a distributed monolith antipattern.

I have run all 3 at big scale. Kafka is still great as long as everyone using it understands it's a stream, not a queue and using it like a queue is going to get them burnt. I don't touch RabbitMQ with a 30ft pole anymore, too many lost days or nights to split brains and other chaos.

Pulsar has mostly replaced Kafka for me because I don't need to worry about people coming along and changing requirements after the fact and saying, actually yeah we do need queue semantics. Or actually yeah we need data larger than the ~48hrs we want to store in Kafka and we don't want to teach our application to read from the archive.

Pulsar is definitely greener than Kafka in a lot of ways but the underlying stuff is very solid, BookKeeper in particular is tough so you aren't really at risk of data loss but you might run into bugs that make brokers do silly things and that can be annoying. Generally speaking though if you validate both the paths you are using and new releases it's been fairly OK to me.

The big thing was being able to directly connect external clients into Pulsar using the Websocket listener on the proxy and plugging in my own authz/authn logic. The eliminated a layer that would otherwise need to be implemented separately.

So far I have been happy with Pulsar, if you haven't tried it for a while you should give it another go. It will only get mature if people use it. :)

> distributed monolith antipattern

Beauty is in the eye of the beholder, I guess. (I would definitely not consider a distributed monolith an antipattern)

Also wanted to add NSQ to your very good list.