Queue Despair: Ordering and Poison Messages
openmymind.net
Queue Despair: Ordering and Poison Messages
1–10 of 49 posts
Re: Queue Despair: Ordering and Poison Messages
#2Ordering is an expensive property.
I built a system where
1. Sensor events were picked up by a ZWave device connected to Samsung SmartThings
2. SmartThings would call a AWS lambda function I wrote (SmartThings lives in AWS so this is efficient)
3. My lambda function posts an event to an SQS queue
4. My home server takes the event off SQS and posts it to RabbitMQ
5. A queue listener takes events from RabbitMQ and takes an action
Re: Queue Despair: Ordering and Poison Messages
#3Rejecting a poison message explicitly IS sufficiently processing it.
It's common to have windows in time where two or more sides may not have agreed what happened before they lose communication. Most of the problem can be solved by idempotency, so when the peer retries, the receiver understands it is looking at a duplicate transaction and can discard it indicating that it succeeded.
Re: Queue Despair: Ordering and Poison Messages
#4Ordering is an expensive property. I built a system where 1. Sensor events were picked up by a ZWave device connected to Samsung SmartThings 2. SmartThings would call a AWS lambda function I wrote (SmartThings lives in AWS so this is efficient) 3. My lambda function posts an event to an SQS queue 4. My home server takes the event off SQS and posts it to RabbitMQ 5. A queue listener takes events from RabbitMQ and take…
But why would the signal to switch on the light ever leave the building?
Re: Queue Despair: Ordering and Poison Messages
#5Ordering is an expensive property. I built a system where 1. Sensor events were picked up by a ZWave device connected to Samsung SmartThings 2. SmartThings would call a AWS lambda function I wrote (SmartThings lives in AWS so this is efficient) 3. My lambda function posts an event to an SQS queue 4. My home server takes the event off SQS and posts it to RabbitMQ 5. A queue listener takes events from RabbitMQ and take…
That is a good example. Off topic, but what is the reason to post the event to SQS and then to RabbitMQ? Why not take the events from SQS directly and take the action?
Re: Queue Despair: Ordering and Poison Messages
#6Ordering is too expensive. Don't ever count on it when using an asynchronous queue. It's akin to storing session in a cache -- you're mixing your metaphors.
A queue should NEVER drop messages - otherwise it's a shit queue. Or you have a bug in your application code that needs to be fixed.
Poison messages are DEFINITELY A SMELL. This means you essentially have a broken interface contract. The code that is adding messages is expecting one thing -- code that is processing messages is expecting a different thing. It needs to be fixed by clearly documenting your message queue expectations and fixing your code. Most likely you need to add clear expectations for the lifetime of a message.
Re: Queue Despair: Ordering and Poison Messages
#7Ordering is an expensive property. I built a system where 1. Sensor events were picked up by a ZWave device connected to Samsung SmartThings 2. SmartThings would call a AWS lambda function I wrote (SmartThings lives in AWS so this is efficient) 3. My lambda function posts an event to an SQS queue 4. My home server takes the event off SQS and posts it to RabbitMQ 5. A queue listener takes events from RabbitMQ and take…
But why would the signal to switch on the light ever leave the building?
[deleted]
Re: Queue Despair: Ordering and Poison Messages
#8RabbitMQ has poison message handling in Quorum queues [0]. They are also FIFO.
[0] https://www.rabbitmq.com/quorum-queues.html#poison-message-h...
Re: Queue Despair: Ordering and Poison Messages
#9My understanding that not losing any messages and strict ordering correspond to "exactly once" delivery which is not possible in general case.
Re: Queue Despair: Ordering and Poison Messages
#10Apart from your unit tests, there is no such thing as "Messages do need to be strictly ordered and messages cannot be lost". You can WISH for messages to come in the right sequence, and even count on it in terms of optimization, but if an event tracks something that happened, and that event comes late - or after a week - your system cannot say "Too bad, I told you, only events in the right order here" (which, at this point, is invalid in itself, as you missed some).