Redis isn't an option?
Or postgres. If you are under 1000 messages a second it functions well as a transactional queue.
RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
111–120 of 173 posts
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#112seems like there is a missing option?
- Message retention (like Kafka)
- Easy consumer scale out (like RabbitMQ)
- No particular ordering guarantees (like RabbitMQ)
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#113I’m personally a fan of Kafka. I think the design of persisting the messages, and tracking offsets for progress instead of message acknowledgments is a much cleaner and more versatile design. You can get all the same advantages of message acknowledgments, but now you can also replay queues, let different applications use the messages (handy for cross cutting event/notification systems) and you get better scaling prop…
> now you can also replay queues yeahnah, that leads to people treating queues like databases (I'm looking at you new york times, you know what you did wrong) its either a queue, or a pubsub, either way its ephemeral. Once its gone, it should stay gone. thats what database, object stores or filesystems are for. Kafka is a beast, has lots of bells and whistles and grinds to a halt when you look at it funny. Yes, it ca…
What would you choose today ?
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#114Earlier quoted context omitted.
> One is smart messaging; dumb clients. The other is dumb messaging; smart clients. All the smartness of the messaging can be implemented in the smart clients. Then you can expose that as a smart messaging api to dumb clients. The most obvious example is kafka streams which exposes a "simple" api rather than dealing directly with kafka, but obviously you could create a less featurefull wrapper than that.
> All the smartness of the messaging can be implemented in the smart clients. How do you do, for example, a queue with priorities client side without it being insanity? That's a relatively basic AMQP thing. Or managing the number of redeliveries for a message that's being repeatedly rejected. You can absolutely try to build some of this with a look-aside shared data store that all clients have to depend on in order t…
Those requirements would definitely be examples of those that are fulfilled by smart messaging.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#115I've seen Tibco Rendezvous used in manufacturing. ~300 megabytes per hour of raw log generated 24/7/365 by tools and control systems in a factory setting. Probably on the order of 10k+ participants in the pub/sub network. If you are running something like a factory where thousands of independent systems need to communicate in some way, this kind of tech starts to look like the only option. If you are orchestrating th…
Our prod cluster generates that about every minute at O(1M) qps. We JUST turned on remote Logs because until now Kafka didn't have capacity.
My previous company's Kafka cluster was handing 20 million messages per second 5 years ago, and dozens of petabytes of data per day. Maybe your particular cluster that didn't have the capacity to handle 1M qps, but Kafka easily had that capacity years ago.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#116Earlier quoted context omitted.
> The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consu…
You've made very good points about smart clients, but at some point one has to ponder if it's worth it or one should just not use kafka in the first place. I've seen databases used as messaging queues and if it was up to me, I'd never do that. It's usually "but we already have kafka + db, why burden ourselves with another messaging technology?", which is fair. > You can stream messages into a buffer and process them…
Completely agree. Kafka was another team's decision, not mine, so I had to figure it out. RabbitMQ is very convenient in that you don't need to read a couple of books on reliable data integration patterns to get something working simply and intuitively.
I am fond of Kafka now that I understand it, but I was also an assembly language programmer in a past life so my opinion is probably in the minority.
Regarding the buffer size: you need to implement back pressure, especially if you are CPU and not IO bound; it's another thing that's easy to get wrong with Kafka.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#117Earlier quoted context omitted.
NATS is something else, but it's awesome. It has awesome throughput and latency out of the box (without Jetstream), while using little resources. I'd recommend considering it, especially as an alternative to RabbitMQ.
I only tested NATS using JetStream and I struggled with the throughput in Python. I probably used it wrong. But your comment may imply that jetstream is slow.
I’m not familiar with the Python lib but it could be waiting for streams to acknowledge each message reception/persistence before sending the next one. Some clients allow transactions to run in parallel e.g. with futures.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#118Earlier quoted context omitted.
> now you can also replay queues yeahnah, that leads to people treating queues like databases (I'm looking at you new york times, you know what you did wrong) its either a queue, or a pubsub, either way its ephemeral. Once its gone, it should stay gone. thats what database, object stores or filesystems are for. Kafka is a beast, has lots of bells and whistles and grinds to a halt when you look at it funny. Yes, it ca…
> (I'm looking at you new york times, you know what you did wrong) You're going to have to be a tiny bit more specific here. NYT is THE factory of wrongness for sure. In every dimension. Are we talking "yellow cake" wrong, or somewhere else on the severity of f'up scale...
They wrote a post on how they disabled the deletion and compaction of the data in Kafka and used it as the source of truth.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#119Earlier quoted context omitted.
NATS is something else, but it's awesome. It has awesome throughput and latency out of the box (without Jetstream), while using little resources. I'd recommend considering it, especially as an alternative to RabbitMQ.
I only tested NATS using JetStream and I struggled with the throughput in Python. I probably used it wrong. But your comment may imply that jetstream is slow.
I didn’t mean to imply that Jetstream is slow. It’s just that I did my benchmarks without it. On a local PC, with 10 KiB messages sent (synchronously) in a loop, I could transfer 3.2 GiB over 5 seconds with 0.2 nanoseconds latency. Performing the same test with RabbitMQ, I got even better throughput out of the box, but way worse latency.
Those numbers are for server 2.9.6 and .NET client 1.0.8.
Re: RabbitMQ vs. Kafka – An Architect’s Dilemma (Part 1)
#120What these articles always miss is what are the real-world scenarios that use Kafka or RabbitMQ? I've never used one professionally and beyond the async processing cases I come across in web dev (mostly sending emails asynchronously), or seeing devs using one to handle 5 req/s which doesn't need it, I don't have a good feel for when they are really needed - especially for companies at sub-Uber scale.
What real-world cases would you use them for?