Redis isn't an option?
I'd also like someone with experience to contrast redis for these use cases.
RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
51–60 of 173 posts
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#52Message ordering is an illusion. Unless you track/store the messages on the client and are willing to deal with stuck queues due to failures in one "poisoned" message.
Couldn't agree more, messages should be completely agnostic from one-another. If you have a decent event-driven architecture, you don't need kafka. and you can be happy with Redis or RabbitMQ
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#53Nice post! RabbitMq is battle tested, exceptionally fast and low resources app. Capable of handling millions of transactions/second. RabbitMQ will handle vast majority of usecases. I'm puzzled why often startups, or even banks use Kafka, soley because is hype. Kafka on the order hand requires massive CPUs, Memory, often requiring its own K8S cluster just to be alive.
If your have a clean event-driven architecture, ie messages are completely agnostic and decoupled from one-another you don't need Kafka.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#54Earlier quoted context omitted.
> One is smart messaging; dumb clients. The other is dumb messaging; smart clients. All the smartness of the messaging can be implemented in the smart clients. Then you can expose that as a smart messaging api to dumb clients. The most obvious example is kafka streams which exposes a "simple" api rather than dealing directly with kafka, but obviously you could create a less featurefull wrapper than that.
> All the smartness of the messaging can be implemented in the smart clients. How do you do, for example, a queue with priorities client side without it being insanity? That's a relatively basic AMQP thing. Or managing the number of redeliveries for a message that's being repeatedly rejected. You can absolutely try to build some of this with a look-aside shared data store that all clients have to depend on in order t…
I totally agree that you cant do a lot of AMQP stuff. As you noted, you can build some of it by managing state via transactional producers, etc - but you definitely cant do everything. The biggest gripe for me is actually dynamic "queue" creation, patterns for topics, etc. So I use an MQ for an MQ ;)
I'm just saying you can "dumb down" the client side on kafka by creating an abstraction layer (or one of the many higher level libs that already do that).
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#55> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…
> who can't just now pluck messages off a queue to process The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. If a certain message processing takes very long, all other messages in that partition will have to wait. Also, with kafka, the max read concurrency is equal to the number of partitions, for something like rabbitMq it is mu…
> The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition.
Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consumer.
RabbitMQ says "let me track this for you", Kafka says "you already need to track this so why duplicate the data in the cluster and complicate the protocol".
If you don't have reliable persistent storage available and insist on using the Kafka cluster to track offsets, you can track processed offsets in memory and whenever your lowest processed offset moves forward, you have your consumer commit that offset manually as part of its message loop.
If your service restarts your downstream commands need to be idempotent of course because you will reconsume messages you may have previously processed, but this would be the case regardless of Kafka or RabbitMQ unless you're using distributed transactions (yuck).
> If a certain message processing takes very long, all other messages in that partition will have to wait.
You can stream messages into a buffer and process them in parallel, and commit the low watermark offset whenever it changes, as described above. I've implemented this in .NET with Channels and saturate the CPUs with no problem.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#56Earlier quoted context omitted.
No, I’m not, because it was years ago, and I’m asking for clarification because what was said immediately sounded wrong to me (I’ve managed a lot of rabbitmq deployments) and you’ve not really given one other than an appeal to authority. guess I have my answer. Can’t find anything that suggests rabbitmq natively supports anything like sink connectors. thanks.
> I am not an expert in either and have only worked with Kafka. > I’ve managed a lot of rabbitmq deployments ... ?
You do not need to be an expert in something's internal workings to manage/monitor a deployment. Surely this does not need to be explained further.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#57Earlier quoted context omitted.
No, I’m not, because it was years ago, and I’m asking for clarification because what was said immediately sounded wrong to me (I’ve managed a lot of rabbitmq deployments) and you’ve not really given one other than an appeal to authority. guess I have my answer. Can’t find anything that suggests rabbitmq natively supports anything like sink connectors. thanks.
> Can’t find anything that suggests rabbitmq natively supports anything like sink connectors Kafka doesnt natively support them either. That would be Kafka Connect. I guess you could use it as an MQ, but it wouldnt be a very good one. Its more used as a data integration platform. If you want more MQ-like functionality OOTB on top of Kafka you would want to use something like Kafka Streams instead.
Thanks for this clarification, this is what I was after.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#58Earlier quoted context omitted.
No, I’m not, because it was years ago, and I’m asking for clarification because what was said immediately sounded wrong to me (I’ve managed a lot of rabbitmq deployments) and you’ve not really given one other than an appeal to authority. guess I have my answer. Can’t find anything that suggests rabbitmq natively supports anything like sink connectors. thanks.
> I am not an expert in either and have only worked with Kafka. > I’ve managed a lot of rabbitmq deployments ... ?
A classic exchange on the internet.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#59Earlier quoted context omitted.
NATS is something else, but it's awesome. It has awesome throughput and latency out of the box (without Jetstream), while using little resources. I'd recommend considering it, especially as an alternative to RabbitMQ.
I only tested NATS using JetStream and I struggled with the throughput in Python. I probably used it wrong. But your comment may imply that jetstream is slow.
I think sometimes the client bindings are/were in need of improvement.
As an example, the C# API was originally very 'go-like' and written to .NET Framework, didn't take advantage of a lot of newer features... to the point a 3rd party client was able to get somewhere between 3-4x the throughput. This is now finally being rectified with a new C# client, however it wouldn't surprise me if other languages have similar pains.
I haven't tested JetStream but my general understanding is that you do have to be mindful of the different options for publishing; especially in the case of JetStream, a synchronous publish call can be relatively time consuming; it's better to async publish and (ab)use the future for flow control as needed.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#60One is a tomato, the other is an orange. From a distance they might look alike but they really are two completely different tools. This is a pretty solid explanation of the differences with good illustrations. Rabbit can do everything Kafka does - and much more - in a more configurable manner. Kafka is highly optimized for essentially one use case and does that well. Nothing in life is free, there are trade-offs ever…
Rabbit is an arse to scale past one broker. It was possible, but a pain, that might have changed. Kafka is just a pain full stop.
At a previous company, about 10 years ago, we had roughly 10 RabbitMQ instances (brokers), all isolated. The system was essentially partitioned by queue server. We had a directory-ish service that would associate clients with their assigned node. It worked well, except if a client got too large we might have to move them to another queue.