Queue Despair: Ordering and Poison Messages • Hacker News Board

Re: Queue Despair: Ordering and Poison Messages

#11

Post by JensRantil » Wed, Dec 08, 2021, 3:04 PM UTC

danesparza wrote: post #6

Ordering is too expensive. Don't ever count on it when using an asynchronous queue. It's akin to storing session in a cache -- you're mixing your metaphors. A queue should NEVER drop messages - otherwise it's a shit queue. Or you have a bug in your application code that needs to be fixed. Poison messages are DEFINITELY A SMELL. This means you essentially have a broken interface contract. The code that is adding messa…

...it might be a smell, but they sure are bound to happen eventually; Corrupt user state or unhanded corner cases etc. Better plan for that in advance.

Re: Queue Despair: Ordering and Poison Messages

#12

Post by cheradenine_uk » Wed, Dec 08, 2021, 3:24 PM UTC

Yes, this is a hard problem. Not even partitioning by tenant will always help you.

This is fundamentally equivalent to database ACID constraints, and the other modes described are great in the same way that if you're able to relax some of the ACID constraints in your code (say, by not being SERIALIZABLE), you get in return nice things (like reads never deadlocking).

If you can't know if message N will change the outcome of processing message N+M, then you have to resolve that before you can proceed - as surely as a serializable database will wait for the outcome of transaction N before being able to proceed with N+M.

Re: Queue Despair: Ordering and Poison Messages

#13

Post by matsemann » Wed, Dec 08, 2021, 3:30 PM UTC

> Even if we ignore poison messages, strict ordering on its own isn't that easy to pull off. Namely, you're limited to a single consumer and can only fetch one message at a time.

Yup. If you want to be sure, you need to persist which message for each entity X you have already processed and ignore older ones. And also make sure you handle race conditions where both messages are handled almost at the same time at two different consumers, by using a lock in a db or so. Which both are annoying, ideally I could just process messages without any care.

Spent all day trying to architecture this for a new queue where ordering matters but we need lots and lots of consumeres working at the same time prefetching messages. My case is actually a bit similar to the one in the article about a "stream of vehicle positions". I only really care about the latest one. Problem is it's hard to know which is the latest one without having a db and check if I've already processed something more recent. Any other ideas on how to efficiently solve this? As in, strategies to handle messages arriving out of order, so I can avoid that as a requirement.

Re: Queue Despair: Ordering and Poison Messages

#14

Post by azdle » Wed, Dec 08, 2021, 3:34 PM UTC

BiteCode_dev wrote: post #4

PaulHoule wrote: post #2

Ordering is an expensive property. I built a system where 1. Sensor events were picked up by a ZWave device connected to Samsung SmartThings 2. SmartThings would call a AWS lambda function I wrote (SmartThings lives in AWS so this is efficient) 3. My lambda function posts an event to an SQS queue 4. My home server takes the event off SQS and posts it to RabbitMQ 5. A queue listener takes events from RabbitMQ and take…

But why would the signal to switch on the light ever leave the building?

Because SmartThings was born out of a software consultancy that knew how to do backend cloud services, but not embedded. The v1 hub was just a pic microcontoller that sent raw radio messages up to the cloud for all parsing and processing.

Re: Queue Despair: Ordering and Poison Messages

#15

Post by tlarkworthy » Wed, Dec 08, 2021, 3:42 PM UTC

I remember trying to productionize an ordered service and the SRE's were banging on about messages-of-death. Their band aid solution were isolated regions. i.e. DO NOT LET YOUR REGIONAL SERVICES COMMUNICATE. What they were worried about were global cascading failures, if/when someone pushes a mistake to prod.

It's kind of a shitty solution to the problem but there you have it, maybe it's the best that can be done. Rollout code changes gradually in individual regions and make sure a bug doesn't bring everything down.

Re: Queue Despair: Ordering and Poison Messages

#16

Post by politician » Wed, Dec 08, 2021, 4:04 PM UTC

Typically with busted messages, an early and easy thing to do is to just shunt them off into a "dead-letter queue". That's just the name of another queue where messages are manually handled.

Re: Queue Despair: Ordering and Poison Messages

#17

Post by deathanatos » Wed, Dec 08, 2021, 4:06 PM UTC

helge9210 wrote: post #9

My understanding that not losing any messages and strict ordering correspond to "exactly once" delivery which is not possible in general case.

You'd want / it would be an "at least once" delivery. If you need to restart delivery due to some issue, the consumer might see a message twice; it's not out of order per se, it's just that the delivery/queue system doesn't know for sure whether it got delivered, and is thus redelivering.

The consumer, of course, must be aware of & ready to handle multiple deliveries. (E.g., you keep a "log position" which represents where you've processed the incoming messages up to.) But if you need ordering+not losing messages, it's the mode you need. (Since "at most once" implies "sometimes losing messages".)

Re: Queue Despair: Ordering and Poison Messages

#18

Post by andoma » Wed, Dec 08, 2021, 4:18 PM UTC

PaulHoule wrote: post #2

Ordering is an expensive property. I built a system where 1. Sensor events were picked up by a ZWave device connected to Samsung SmartThings 2. SmartThings would call a AWS lambda function I wrote (SmartThings lives in AWS so this is efficient) 3. My lambda function posts an event to an SQS queue 4. My home server takes the event off SQS and posts it to RabbitMQ 5. A queue listener takes events from RabbitMQ and take…

If there only was another way to turn on a lamp…

Re: Queue Despair: Ordering and Poison Messages

#19

Post by PaulHoule » Wed, Dec 08, 2021, 4:24 PM UTC

tbonesteaks wrote: post #5

PaulHoule wrote: post #2

Ordering is an expensive property. I built a system where 1. Sensor events were picked up by a ZWave device connected to Samsung SmartThings 2. SmartThings would call a AWS lambda function I wrote (SmartThings lives in AWS so this is efficient) 3. My lambda function posts an event to an SQS queue 4. My home server takes the event off SQS and posts it to RabbitMQ 5. A queue listener takes events from RabbitMQ and take…

That is a good example. Off topic, but what is the reason to post the event to SQS and then to RabbitMQ? Why not take the events from SQS directly and take the action?

Events that originate from inside the house go into the RabbitMQ. RabbitMQ is the central bus for all sorts of things. In the process of diagnosing that latency I found that part was pretty fast.

Re: Queue Despair: Ordering and Poison Messages

#20

Post by PaulHoule » Wed, Dec 08, 2021, 4:27 PM UTC

azdle wrote: post #14

BiteCode_dev wrote: post #4

Earlier quoted context omitted.

But why would the signal to switch on the light ever leave the building?

Because SmartThings was born out of a software consultancy that knew how to do backend cloud services, but not embedded. The v1 hub was just a pic microcontoller that sent raw radio messages up to the cloud for all parsing and processing.

(1) If you are letting people code their own event handlers you wouldn't trust them to run them on a tiny machine like that.

(2) A system like that probably wants to be able to respond to events both inside and outside the house. For instance, turn on your lights remotely with a phone. A cloud component is the reliable way to do that.

(3) At the time I couldn't find decent Zigbee or ZWave hubs other than the SmartThings hub. The cloud dependence is silly, but other than that the hub is great and connects to almost everything. I could go with a Kickstarter hack or try to roll my own but I don't think it would be much better.