31–40 of 173 posts
So use RabbitMQ.
I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design. You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling prop…
Maybe 95% of cases, but not all.
Long message processing time really kills kafka in a way it doesn't kill Rabbit Mq. Combine it with inherent read paralelism being limited to the number of partitions. Add in high variability of message rates and bingo, that's like 90% of the issues I've had with kafka over the years.
Earlier quoted context omitted.
> parse and sanitize some extremely dirty, unstructured data and pass it along somewhere else can you be more specific? that to me sounds like hello world for either of these tools. "santize data" is an application level concern that neither rabbit or kafka would handle. as far as "pass along somewhere else" again both tools can do.
It was a Sink Connector. I don’t know what it was or wasn’t supposed to do but I was asked to do it, as is often the case in tech. I could have done any number of transformations in that process though, which I’m not sure rabbitmq supports
I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design. You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling prop…
> You can get all the same advantages of message acknowledgments, but now you can also replay queues with rmq you can reject/nack a message and have it put back on the queue. rmq is not well suited for long term historical retention inside queues a-la kafka's logs but it is possible to do. > let different applications use the messages (handy for cross cutting event/notification systems) rmq also does a publish once a…
Kafka is really nice if you don't care that much about latency during peak load and you don't have absurd processing times for messages.
NATS ( https://nats.io/ ) is another option, though I'm not sure if it's still considered a viable Kafka replacement.
NATS is something else, but it's awesome. It has awesome throughput and latency out of the box (without Jetstream), while using little resources. I'd recommend considering it, especially as an alternative to RabbitMQ.
> one is a message broker, and the other is a distributed streaming platform I think this is an odd way of putting it. One is smart messaging; dumb clients. The other is dumb messaging; smart clients. It turns out the latter (i.e. Kafka) scales wonderfully so you can send more data, but you add complexity to your clients, who can't just now pluck messages off a queue to process, or have messages retry upon the first…
> One is smart messaging; dumb clients. The other is dumb messaging; smart clients. All the smartness of the messaging can be implemented in the smart clients. Then you can expose that as a smart messaging api to dumb clients. The most obvious example is kafka streams which exposes a "simple" api rather than dealing directly with kafka, but obviously you could create a less featurefull wrapper than that.
How do you do, for example, a queue with priorities client side without it being insanity? That's a relatively basic AMQP thing. Or managing the number of redeliveries for a message that's being repeatedly rejected.
You can absolutely try to build some of this with a look-aside shared data store that all clients have to depend on in order to emulate having the capability in the broker, but you just introduced another common point of failure in addition to the messaging infrastructure. Life is too short for this.
One is a tomato, the other is an orange. From a distance they might look alike but they really are two completely different tools. This is a pretty solid explanation of the differences with good illustrations. Rabbit can do everything Kafka does - and much more - in a more configurable manner. Kafka is highly optimized for essentially one use case and does that well. Nothing in life is free, there are trade-offs ever…
Kafka is just a pain full stop.
Earlier quoted context omitted.
It was a Sink Connector. I don’t know what it was or wasn’t supposed to do but I was asked to do it, as is often the case in tech. I could have done any number of transformations in that process though, which I’m not sure rabbitmq supports
It sounds to me like you aren't really even sure what you built. I have operated both rabbit and kafka at scale I definitely do not have it backwards :)
NATS ( https://nats.io/ ) is another option, though I'm not sure if it's still considered a viable Kafka replacement.
it is simple to configure, has good documentation, and excellent integration into most languages. It guarantees uptime, and thats about it. It clusters really well, so you can swap out instances, or scale in/out as you need.
Earlier quoted context omitted.
It sounds to me like you aren't really even sure what you built. I have operated both rabbit and kafka at scale I definitely do not have it backwards :)
No, I’m not, because it was years ago, and I’m asking for clarification because what was said immediately sounded wrong to me (I’ve managed a lot of rabbitmq deployments) and you’ve not really given one other than an appeal to authority. guess I have my answer. Can’t find anything that suggests rabbitmq natively supports anything like sink connectors. thanks.
> I’ve managed a lot of rabbitmq deployments
... ?