151–160 of 173 posts
Message ordering is an illusion. Unless you track/store the messages on the client and are willing to deal with stuck queues due to failures in one "poisoned" message.
There are different kinds of order. Yes, there’s no total order in a distributed system, but you can have certain partial order guarantees. It’s nice if something is added before it’s updated, for instance.
Earlier quoted context omitted.
> The problem is you cannot mark individual messages as read, for a given consumer&partition you can only update the offset for a partition. Hence "smart clients". If you MUST process every message at least once, you will anyway be tracking messages individually on the client (e.g. a DB or file system plus logic for idempotent message processing) and thus disable auto-offset commits back to the cluster for your consu…
If you have idempotent messages, why can't you use auto offset committing?
Earlier quoted context omitted.
There are different kinds of order. Yes, there’s no total order in a distributed system, but you can have certain partial order guarantees. It’s nice if something is added before it’s updated, for instance.
Could you expand on that? How would you achieve "partial order" guarantees?
If a consumer sees an event B, it will have certainly have seen the event A before that.
Assuming business logic is correctly written, that saves you from having to write certain retry logic on the B handlers. This requires the message queue to be always available. If it goes down, the system would not make progress (like a db - in fact the MQ is a db).
Once you add more actors/nodes to the same related events, maintaining a “causal order” can be very tricky and subtle, especially if you have an MQ and a DB as multiple sources of truth. So I’m not exactly endorsing it, even though MQ-as-a-DB (aka event sourcing) is a very interesting idea.
Earlier quoted context omitted.
There are different kinds of order. Yes, there’s no total order in a distributed system, but you can have certain partial order guarantees. It’s nice if something is added before it’s updated, for instance.
Could you expand on that? How would you achieve "partial order" guarantees?
That doesn't fix your example of dealing with individual errors, but in many cases that's enough.
Earlier quoted context omitted.
It’s true FOSS, and the server is standalone Go binary that’s so small it can even be embedded. Lots of language bindings for clients. Has persistence, durability, and nicely aligns into a raft-like cluster in a DC without a separate orchestrator. I’m a big fan – never understood why it’s not at the top of the list in these tech reviews.
Rabbitmq is FOSS, has lots of language bindings. It has persistence, durability, and doesn't require a separate orchestrator.
Are the horizontal scaling issues solved now?
Earlier quoted context omitted.
I have run all 3 at big scale. Kafka is still great as long as everyone using it understands it's a stream, not a queue and using it like a queue is going to get them burnt. I don't touch RabbitMQ with a 30ft pole anymore, too many lost days or nights to split brains and other chaos. Pulsar has mostly replaced Kafka for me because I don't need to worry about people coming along and changing requirements after the fac…
I am contemplating this exact topic for my project at this moment. It would be great if you can briefly explain what, as per your understanding, stream vs queue semantic are. I am studying it and got somewhat confusing discussions on the internet and in person forums.
This is a gross oversimplification as all ELI5 are but it's a decent rule regardless.
The reason for this is that streaming systems by and large function on some sort of offset mechanism. Your client when it's receiving messages is generally calling something similar to poll(fromOffset, max) to get some messages and then keeping track of the max offset it's published somewhere (Kafka has consumer groups to help you store your offsets).
The problem with this model is you can only generally a) get messages in order on a given topic partition starting from some offset and b) you can only "commit" the latest message you processed.
This is fine if the chance of failure for a given message is the same for all messages. i.e streaming database updates into an backup. Either the backup target is available or it's not, if one message fails all would likely fail.
On the other end of the spectrum you have something like a queue of webhook jobs to execute against 3rd party/user supplied targets. The chance of any given webhook failing is entirely divorced from the rest in the queue.
So if you were to try use a stream for the webhook case you would quickly get blocked on the first bad webhook server you ran into. While with a proper queueing system you could kick that job back with a delay and process it again later without blocking work on other tasks or being able to commit which tasks have been processed.
This is generally called head of line blocking problem.
If someone is asking if they should decide between RabbitMQ vs Kafka, they should 100% use RabbitMQ. It means they have no idea what they're dealing with, the architectural differences, and the investment that the company needs in order to use Kafka. So use RabbitMQ.
Kafka is overkill in most scenarios and you should probably just see if postgres isn't enough for your needs first (especially since you will almost certainly already need a database anyway). Kafka is more pain to setup and run than it ought to be. But underlying it is a useful and sensible abstraction for building robust distributed systems.
Earlier quoted context omitted.
No, I’m not, because it was years ago, and I’m asking for clarification because what was said immediately sounded wrong to me (I’ve managed a lot of rabbitmq deployments) and you’ve not really given one other than an appeal to authority. guess I have my answer. Can’t find anything that suggests rabbitmq natively supports anything like sink connectors. thanks.
So let me get this straight. You've used Kafka once, RabbitMQ never. You don't really know what you did with Kafka. But you somehow know that RabbitMQ cannot do the thing which you don't really remember anymore. Doesn't make much sense to be honest. Nobody can really have any sources for RabbitMQ being able to do it if you don't know what it supposedly cannot do. The way you descibed it, is that you simply read data…
Not true, and some also for the rest of your snotty comment, I'd have a response but it's best not to engage trolls. Another commenter answered the question I had. Good luck.
Also, did you register solely to make this comment? Pretty sad display, really.
Earlier quoted context omitted.
My previous company's Kafka cluster was handing 20 million messages per second 5 years ago, and dozens of petabytes of data per day. Maybe your particular cluster that didn't have the capacity to handle 1M qps, but Kafka easily had that capacity years ago.
I have to ask, what value is this adding business-wise to store so much?
I'm sure there remain good use cases for message buses that you have to run yourself, where there really are millions of messages that can't be batched up and need real-time and whatnot. But you can get pretty far with: 1. Write a bunch of records to an S3 object. 2. Trigger a lambda to process — infinite scale out! and if a queue really is needed due to constrained consumers, then: 1. Write a bunch of records to an…