Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
1–10 of 82 posts
Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
#2Biggest thing to watch out with this approach is that you will inevitably have some failure or bug that will 10x, 100x, or 1000x the rate of dead messages and that will overload your DLQ database. You need a circuit breaker or rate limit on it.
Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
#3Biggest thing to watch out with this approach is that you will inevitably have some failure or bug that will 10x, 100x, or 1000x the rate of dead messages and that will overload your DLQ database. You need a circuit breaker or rate limit on it.
This! Only thing worse than your main queue backing off is you dropping items from going into the DLQ because it can’t stay up.
Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
#4Biggest thing to watch out with this approach is that you will inevitably have some failure or bug that will 10x, 100x, or 1000x the rate of dead messages and that will overload your DLQ database. You need a circuit breaker or rate limit on it.
If you can’t deliver to the DLQ, then what? Then you’re missing messages either way. Who cares if it’s down this way or the other?
Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
#5Biggest thing to watch out with this approach is that you will inevitably have some failure or bug that will 10x, 100x, or 1000x the rate of dead messages and that will overload your DLQ database. You need a circuit breaker or rate limit on it.
If you can’t deliver to the DLQ, then what? Then you’re missing messages either way. Who cares if it’s down this way or the other?
Sure, but you still need to design around this problem. It’s going to be a happy accident that everything turns out fine if you don’t.
Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
#6Biggest thing to watch out with this approach is that you will inevitably have some failure or bug that will 10x, 100x, or 1000x the rate of dead messages and that will overload your DLQ database. You need a circuit breaker or rate limit on it.
If you can’t deliver to the DLQ, then what? Then you’re missing messages either way. Who cares if it’s down this way or the other?
Not necessarily. If you can't deliver the message somewhere you don't ACK it, and the sender can choose what to do (retry, backoff, etc.)
Sure, it's unavailability of course, but it's not data loss.
Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
#7> FOR UPDATE SKIP LOCKED
Learned something new today. I knew what FOR UPDATE did, but somehow I've never RTFM'd hard enough to know about the SKIP LOCKED directive. Thats pretty cool.
Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
#8Biggest thing to watch out with this approach is that you will inevitably have some failure or bug that will 10x, 100x, or 1000x the rate of dead messages and that will overload your DLQ database. You need a circuit breaker or rate limit on it.
If you can’t deliver to the DLQ, then what? Then you’re missing messages either way. Who cares if it’s down this way or the other?
The point is to not take the whole server down with it. Keeps the other applications working.
Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
#9Another day, another “Using PostgreSQL for…” thing it wasn’t designed for. This isn’t a good idea. What happens when the queue goes down and all messages are dead lettered? What happens when you end up with competing messages? This is not the way.
Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems
#10Segment uses MySQL as queue not even as DLQ. It works at their scale. So there are many (not all) systems that can tolerate this as queue.
I have simple flow: tasks are order of thousands an hour. I just use postgresql. High visibility, easy requeue, durable store. With appropriate index, it’s perfectly fine. LLM will write skip locked code right first time. Easy local dev. I always reach for Postgres for event bus in low volume system.