Queue Despair: Ordering and Poison Messages

This is why I prefer putting the "bad" message into a deadletter queue instead of dropping it entirely(and alert on messages going into this queue). This unblocks the queue so it can continue working and allows one to decide if it should be dropped or re-processed.

Sidekiq has a lot of code to deal with both of these issues.

Sidekiq does not guarantee ordering within a queue; that’s a terrible, very expensive guarantee. Developers don’t want total ordering of jobs within a queue, they want to know that Job A will fully execute before Job B. There might be 1000 other jobs in the queue that are completely independent of that ordering but we’ve screwed ourselves by forcing total queue ordering. Instead Sidekiq Pro provides a workflow API, Sidekiq::Batch, which allows the developer to author higher-level workflows for Job A -> Job B which provides the ordering guarantee.

For poison pills, we detect jobs which were running when a Sidekiq process died. If this happens multiple times, the job will be sent to the dead letter queue so the developer can deal with them manually. If they were part of a Batch, the workflow will stall until the developer fixes the issue and executes the job manually to resume the workflow.

Have you considered frameworks like Orleans? Presumably you would have an individual actor “grain” for each vehicle and it would serialise the messages to it for you.

The state of the vehicle is kept in memory for some time.

> Adding a timestamp to each message is an easy way for consumers to discard any out-of-order messages.

Not correct, but it's very easy to think timestamps will solve this. Timestamps aren't good because system times aren't synced across different computers precisely. Meaning if Producer A creates the first event, and Producer B creates a second event 50ms after (imagine a single row gets updated very quickly twice), but the system time on Producer A is 100ms ahead of Producer B and the event from Producer B gets to the consumers first (variable network latency), the event from Producer A will look like the latest event from a timestamp perspective and overwrite the Producer B event.

One way to solve is it to use not use timestamps, but use a monotonically increasing version number associated with a row that gets updated for every event/update or whatever and is sent along with the event message payload. The book, Designing data intensive systems, goes into this problem a whole lot. Recommend it to anyone discussing architecture. Issues like this will seem obvious to you after reading

Introduce processing log messages in order to maintain order and not lose any messages, even a poisoned one. The faulty messages are processed as any other message (write a 'did msg x : faulted') and then go to a fault-queue.

If you have inter-dependence between messages, you need to have a message id scheme that shows the interdependence. For example, a hierarchical message order can have a .dot delimited scheme. If a poison message is a parent, the subsequent message can go to the fault-queue as well.