Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Queue Despair: Ordering and Poison Messages

openmymind.net

Post reply on HN
Print view View original

31–40 of 49 posts

Re: Queue Despair: Ordering and Poison Messages

#31

Post by srazzaque »

ydnaclementine wrote: post #28

> Adding a timestamp to each message is an easy way for consumers to discard any out-of-order messages. Not correct, but it's very easy to think timestamps will solve this. Timestamps aren't good because system times aren't synced across different computers precisely. Meaning if Producer A creates the first event, and Producer B creates a second event 50ms after (imagine a single row gets updated very quickly twice),…

Yep, and there's no straightforward way to know if you missed a message by looking at a timestamp alone. If I receive a message at time "1" then time "3", how will I know I was meant to have seen something at time "2"?

Sequence numbering is how FIX does it. And I think its quite neat that it does this at a _protocol_ level. This means that a FIX client/engine will typically take care of sequence numbering, out-of-order detection (can happen during re-send requests), buffering any ahead-of-time messages, requesting gapfills, etc. It will only present your application layer code with in-order messages.

I'm not aware of any universal pattern for dealing with poison pill messages. Completeness detection and dealing with messages that crash your system are 2 separate concerns.

Re: Queue Despair: Ordering and Poison Messages

#32

Post by gpderetta »

ydnaclementine wrote: post #28

> Adding a timestamp to each message is an easy way for consumers to discard any out-of-order messages. Not correct, but it's very easy to think timestamps will solve this. Timestamps aren't good because system times aren't synced across different computers precisely. Meaning if Producer A creates the first event, and Producer B creates a second event 50ms after (imagine a single row gets updated very quickly twice),…

FWIW GPS clocks and PTP can keep clocks synchronized well below the us.

Re: Queue Despair: Ordering and Poison Messages

#33

Post by mfateev »

I think queues are the wrong abstraction to model business processes. That's why a trivial issue like a non recoverable failure during processing a message becomes such a headache. The same goes for ordering. An orchestrator like temporal.io allows modeling your business use case using higher level abstractions that hide all this low level complexity.

Disclaimer: I'm the tech lead of the temporal.io open source project and the CEO of the affiliated company.

Re: Queue Despair: Ordering and Poison Messages

#34

Post by sam0x17 »

ydnaclementine wrote: post #28

> Adding a timestamp to each message is an easy way for consumers to discard any out-of-order messages. Not correct, but it's very easy to think timestamps will solve this. Timestamps aren't good because system times aren't synced across different computers precisely. Meaning if Producer A creates the first event, and Producer B creates a second event 50ms after (imagine a single row gets updated very quickly twice),…

Yes, this bit me recently. Had to add an atomic index field instead, as some messages were arriving out of order because their timestamps were the same as other messages.

Re: Queue Despair: Ordering and Poison Messages

#35

Post by sam0x17 »

gkop wrote: post #25
User23 wrote: post #23

Earlier quoted context omitted.

> Ordering is too expensive. Don't ever count on it when using an asynchronous queue In a field where precision is absolutely necessary, it's unfortunate to use the term queue to describe something that is not a queue.

Curious, what's your definition of queue?

By definition a queue is FIFO, which requires preserving ordering, or else you could end up with FIRO (first in, random out)

Re: Queue Despair: Ordering and Poison Messages

#36

Post by sam0x17 »

PaulHoule wrote: post #19
tbonesteaks wrote: post #5

Earlier quoted context omitted.

That is a good example. Off topic, but what is the reason to post the event to SQS and then to RabbitMQ? Why not take the events from SQS directly and take the action?

Events that originate from inside the house go into the RabbitMQ. RabbitMQ is the central bus for all sorts of things. In the process of diagnosing that latency I found that part was pretty fast.

One thing I will say is if you need queuing, very few people actually need global queuing, they only need per-user queuing. Have fixed a number of systems where EVERY event in the world was going through the same queue, and replaced that with an array of queues sort of solution. At the end of the day this usually doesn't even need a real queue, just some database transactions and atomic ordering columns that ensure consistency and order of the events within some very specific scope (like a user). If the most events you'll ever see in a row where their order (with respect to each other) matters is like 5-20, you probably don't even need a real queuing service.

Re: Queue Despair: Ordering and Poison Messages

#37

Post by thingification »

One way to deal with this is to divide up your event streams into small streams - say one per order. Those small streams then may be aggregated into a larger stream so that you can just process events for all the orders together, for example.

If you hit a poison message, block just that smaller stream, not the aggregated larger stream. Once you fix the problem, reprocess the entire small stream starting from the poison event, or the next event after that. The "entire" stream here might be just a handful of events.

Greg Young's Event Store (https://www.eventstore.com/) works this way (there's a $by_category projection that produces the aggregated streams).

Caveat: I haven't actually implemented this mechanism because I've been able to get away without it, because we have some legacy event streams that aren't split up in this way, and because nobody else has yet added support for it to the tools I'm using.

Re: Queue Despair: Ordering and Poison Messages

#38

Post by cgio »

mfateev wrote: post #33

I think queues are the wrong abstraction to model business processes. That's why a trivial issue like a non recoverable failure during processing a message becomes such a headache. The same goes for ordering. An orchestrator like temporal.io allows modeling your business use case using higher level abstractions that hide all this low level complexity. Disclaimer: I'm the tech lead of the temporal.io open source proje…

Hiding this complexity is useful if it also means handling it. What are the key patterns you apply in temporal to hide it? I’ve had a look at temporal and find it really interesting.

Re: Queue Despair: Ordering and Poison Messages

#39

Post by bergundy »

cgio wrote: post #38
mfateev wrote: post #33

I think queues are the wrong abstraction to model business processes. That's why a trivial issue like a non recoverable failure during processing a message becomes such a headache. The same goes for ordering. An orchestrator like temporal.io allows modeling your business use case using higher level abstractions that hide all this low level complexity. Disclaimer: I'm the tech lead of the temporal.io open source proje…

Hiding this complexity is useful if it also means handling it. What are the key patterns you apply in temporal to hide it? I’ve had a look at temporal and find it really interesting.

Instead of directly using queues in a Temporal Workflow, the Workflow (which is written with plain code), schedules an Activity that the system is responsible for, behind the scenes, the Activity is just an item put on a queue. Activities have retry policies which are also handled by the system. If an Activity attempt fails and should not retry according to the policy, an exception is thrown in the Workflow to be handled using code.

Using the TypeScript SDK, you can catch that exception here: https://github.com/temporalio/samples-typescript/blob/9d9108...

Re: Queue Despair: Ordering and Poison Messages

#40

Post by legulere »

kimi wrote: post #10

Apart from your unit tests, there is no such thing as "Messages do need to be strictly ordered and messages cannot be lost". You can WISH for messages to come in the right sequence, and even count on it in terms of optimization, but if an event tracks something that happened, and that event comes late - or after a week - your system cannot say "Too bad, I told you, only events in the right order here" (which, at this…

If you want a reliable system it’s very important. Those concepts very much relate to Isolation and Consistency in ACID. Just take a look at different levels at https://jepsen.io/consistency

There are certainly fields with lower correctness requirements where inconsistent data is not such a big deal though.

Post reply on HN