Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Queue Despair: Ordering and Poison Messages

openmymind.net

Post reply on HN
Print view View original

41–49 of 49 posts

Re: Queue Despair: Ordering and Poison Messages

#41

Post by pc86 »

outsomnia wrote: post #3

Rejecting a poison message explicitly IS sufficiently processing it. It's common to have windows in time where two or more sides may not have agreed what happened before they lose communication. Most of the problem can be solved by idempotency, so when the peer retries, the receiver understands it is looking at a duplicate transaction and can discard it indicating that it succeeded.

A message that gets "skipped" - for lack of a better term - because it's already been processed, and a poison message, are not the same thing.

Re: Queue Despair: Ordering and Poison Messages

#42

Post by bob1029 »

srazzaque wrote: post #31
ydnaclementine wrote: post #28

> Adding a timestamp to each message is an easy way for consumers to discard any out-of-order messages. Not correct, but it's very easy to think timestamps will solve this. Timestamps aren't good because system times aren't synced across different computers precisely. Meaning if Producer A creates the first event, and Producer B creates a second event 50ms after (imagine a single row gets updated very quickly twice),…

Yep, and there's no straightforward way to know if you missed a message by looking at a timestamp alone. If I receive a message at time "1" then time "3", how will I know I was meant to have seen something at time "2"? Sequence numbering is how FIX does it. And I think its quite neat that it does this at a _protocol_ level. This means that a FIX client/engine will typically take care of sequence numbering, out-of-ord…

The implication in the FIX use case is that you should have a single writer/consumer of these events, at least for sequencing purposes. Attempting to coordinate sequencing across multiple threads (or computers) will just slow you down. Consider that a CAS or volatile write is ~10x slower on modern x86 than a single thread tearing through the same items without any contention. A 10x slowdown is pretty much the best-case scenario if you must have a contended resource shared between multiple threads. You only lose more orders of magnitude the further you go into this rabbit hole. The LMAX Disruptor is a good example of how we can rethink these sorts of problems and solve them in novel ways. I have linked a document that I think provides a really good perspective on this: [0].

The poison pill case is a business logic thing on either side of the queue. You can either do validation up-front before sticking requests into the queue, or after the fact when processing in batches. Either way, you will ultimately need to be able to handle problems on both sides of the fence. Something that doesn't look like "poison" on the way into the queue could become pretty nasty with certain emergent state as events are processed on the other side.

[0]: https://lmax-exchange.github.io/disruptor/disruptor.html

Re: Queue Despair: Ordering and Poison Messages

#43

Post by lmilcin »

ydnaclementine wrote: post #28

> Adding a timestamp to each message is an easy way for consumers to discard any out-of-order messages. Not correct, but it's very easy to think timestamps will solve this. Timestamps aren't good because system times aren't synced across different computers precisely. Meaning if Producer A creates the first event, and Producer B creates a second event 50ms after (imagine a single row gets updated very quickly twice),…

> One way to solve is it to use not use timestamps, but use a monotonically increasing version number associated with a row that gets updated for every event/update or whatever and is sent along with the event message payload.

It is a concept that is known as vector clock. I suggest take a look at vector and Lamport clocks (named after Leslie Lamport), very useful in distributed systems using messages.

Re: Queue Despair: Ordering and Poison Messages

#44

Post by lmilcin »

mfateev wrote: post #33

I think queues are the wrong abstraction to model business processes. That's why a trivial issue like a non recoverable failure during processing a message becomes such a headache. The same goes for ordering. An orchestrator like temporal.io allows modeling your business use case using higher level abstractions that hide all this low level complexity. Disclaimer: I'm the tech lead of the temporal.io open source proje…

It is a problem only if you are mixing up application layers.

If you keep your queueing system and business process as separate layers with queueing system serving only as a means of transporting business events then you can make it all to work correctly.

Think in terms of IP protocol (as in TCP/IP). It is unsuitable for transmitting financial transactions. Yet, financial transactions can be made to work on top of it if you separate the layers and treat IP only as a component mechanism of getting data from A to B.

Re: Queue Despair: Ordering and Poison Messages

#45

Post by [deleted] »

danesparza wrote: post #6

Ordering is too expensive. Don't ever count on it when using an asynchronous queue. It's akin to storing session in a cache -- you're mixing your metaphors. A queue should NEVER drop messages - otherwise it's a shit queue. Or you have a bug in your application code that needs to be fixed. Poison messages are DEFINITELY A SMELL. This means you essentially have a broken interface contract. The code that is adding messa…

[deleted]

Re: Queue Despair: Ordering and Poison Messages

#46

Post by mfateev »

lmilcin wrote: post #44
mfateev wrote: post #33

I think queues are the wrong abstraction to model business processes. That's why a trivial issue like a non recoverable failure during processing a message becomes such a headache. The same goes for ordering. An orchestrator like temporal.io allows modeling your business use case using higher level abstractions that hide all this low level complexity. Disclaimer: I'm the tech lead of the temporal.io open source proje…

It is a problem only if you are mixing up application layers. If you keep your queueing system and business process as separate layers with queueing system serving only as a means of transporting business events then you can make it all to work correctly. Think in terms of IP protocol (as in TCP/IP). It is unsuitable for transmitting financial transactions. Yet, financial transactions can be made to work on top of it…

I think we are in agreement here. Temporal does exactly what you described. It uses queues to transporting tasks to processes. But it completely hides them from the business process code.

The issue is that 99.9% of developers use queues directly in their business applications.

Re: Queue Despair: Ordering and Poison Messages

#47

Post by rstuart4133 »

ydnaclementine wrote: post #28

> Adding a timestamp to each message is an easy way for consumers to discard any out-of-order messages. Not correct, but it's very easy to think timestamps will solve this. Timestamps aren't good because system times aren't synced across different computers precisely. Meaning if Producer A creates the first event, and Producer B creates a second event 50ms after (imagine a single row gets updated very quickly twice),…

> Not correct, but it's very easy to think timestamps will solve this. Timestamps aren't good because system times aren't synced across different computers precisely.

They can be. That's what Google Spanner does, using GPS and atomic clocks. It's not hard or expensive. An atomic clock + GPS will set you back under $1000 https://www.ebay.com/itm/174750548607 for one in a nice box, $200 for a PCB https://www.ebay.com/itm/353611628534. Apparently it gets you to with 1e-11, which is about 10 pico seconds (I think).

But that doesn't solve the problem. Lets say you have two event producers, and both produce two events at times t and t+1. Once they both arrive, it's trivial to process then in the right order. Your problem is there is an unreliable network connecting you to these producers. Those atomic clocks can guarantee those events t and t+1 are distinguishable events if they are just 10ps apart, but how long do you wait for t to arrive before you decide to process the event at t+1 because there was no event at t. I can absolutely guarantee you whatever time you decide is reasonable, the universe will at some point screw you over and drop the event occurring at on your doorstep just after you decided to process t1.

Your issue isn't that it's difficult, it's is you are living in a state of sin if you believe the problem solvable given the premises.

PS: Google Spanner doesn't attempt to do the impossible. In Spanners case there is a single event producer, and Spanner is "merely" trying to record the event consistently across multiple nodes. If there are multiple event producers then it will serialise them in some order, but if there are two disconnected, independent Spanners out there processing the same events from the same producers Google is not claiming there would decide on the same order. That would need a God more powerful than Google.

Re: Queue Despair: Ordering and Poison Messages

#48

Post by tatersolid »

politician wrote: post #16

Typically with busted messages, an early and easy thing to do is to just shunt them off into a "dead-letter queue". That's just the name of another queue where messages are manually handled.

Now what if the rest of your messages depended on the poison message? Think “inventory in” is the poison message and now all subsequent orders for that product are cancelled.

That’s a toy example; your system corruption could get much worse due to a poison message depending on the application.

Re: Queue Despair: Ordering and Poison Messages

#49

Post by politician »

tatersolid wrote: post #48
politician wrote: post #16

Typically with busted messages, an early and easy thing to do is to just shunt them off into a "dead-letter queue". That's just the name of another queue where messages are manually handled.

Now what if the rest of your messages depended on the poison message? Think “inventory in” is the poison message and now all subsequent orders for that product are cancelled. That’s a toy example; your system corruption could get much worse due to a poison message depending on the application.

In practice this isn’t an issue. You put alarms on that queue and triage it immediately.
Post reply on HN