Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

diljitpr.net

Post reply on HN
Print view View original

11–20 of 82 posts

Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

#11

Post by konart »

xyzzy_plugh wrote: post #6
pletnes wrote: post #4

Earlier quoted context omitted.

If you can’t deliver to the DLQ, then what? Then you’re missing messages either way. Who cares if it’s down this way or the other?

Not necessarily. If you can't deliver the message somewhere you don't ACK it, and the sender can choose what to do (retry, backoff, etc.) Sure, it's unavailability of course, but it's not data loss.

If you are reading from Kafka (for example) and you can't do anything with a message (broken json as an example) and you can't put it into a DLQ - you have not other option but to skip it or stop on it, no?

Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

#12

Post by odie5533 »

reactordev wrote: post #9

Another day, another “Using PostgreSQL for…” thing it wasn’t designed for. This isn’t a good idea. What happens when the queue goes down and all messages are dead lettered? What happens when you end up with competing messages? This is not the way.

You wouldn't ack the message if you're not up to process it.

Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

#13

Post by TexanFeller »

Ofc I wouldn't us it for extremely high scale event processing, but it's great default for a message/task queue for 90% of business apps. If you're processing under a few 100m events/tasks per day with less than ~10k concurrent processes dequeuing from it it's what I'd default to.

I work on apps that use such a PG based queue system and it provides indispensable features for us we couldn't achieve easily/cleanly with a normal queue system such as being able to dynamically adjust the priority/order of tasks being processed and easily query/report on the content of the queue. We have many other interesting features built into it that are more specific to our needs as well that I'm more hesitant to describe in detail here.

Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

#14

Post by hnguyen14 »

reactordev wrote: post #9

Another day, another “Using PostgreSQL for…” thing it wasn’t designed for. This isn’t a good idea. What happens when the queue goes down and all messages are dead lettered? What happens when you end up with competing messages? This is not the way.

How so? There are queues that use SQL (or no-SQL) databases as the persistence layer. Your question is more specific to the implementation, not the database as persistence layer itself. And there are ways to address it.

Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

#15

Post by singron »

konart wrote: post #11
xyzzy_plugh wrote: post #6

Earlier quoted context omitted.

Not necessarily. If you can't deliver the message somewhere you don't ACK it, and the sender can choose what to do (retry, backoff, etc.) Sure, it's unavailability of course, but it's not data loss.

If you are reading from Kafka (for example) and you can't do anything with a message (broken json as an example) and you can't put it into a DLQ - you have not other option but to skip it or stop on it, no?

Generally yes, but if you use e.g. the parallel consumer, you can potentially keep processing in that partition to avoid head-of-line blocking. There are some downsides to having a very old unprocessed record since it won't advance the consumer group's offset past that record, and it instead keeps track of the individual offsets it has completed beyond it, so you don't want to be in that state indefinitely, but you hope your DLQ eventually succeeds.

But if your DLQ is overloaded, you probably want to slow down or stop since sending a large fraction of your traffic to DLQ is counter productive. E.g. if you are sending 100% of messages to DLQ due to a bug, you should stop processing, fix the bug, and then resume from your normal queue.

Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

#16

Post by senbrow »

reactordev wrote: post #9

Another day, another “Using PostgreSQL for…” thing it wasn’t designed for. This isn’t a good idea. What happens when the queue goes down and all messages are dead lettered? What happens when you end up with competing messages? This is not the way.

Criticism without a better solution is only so valuable.

How would you do this instead, and why?

Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

#17

Post by rr808 »

rbranson wrote: post #2

Biggest thing to watch out with this approach is that you will inevitably have some failure or bug that will 10x, 100x, or 1000x the rate of dead messages and that will overload your DLQ database. You need a circuit breaker or rate limit on it.

I worked on an app that sent an internal email with stack trace whenever an unhandled exception occurred. Worked great until the day when there was an OOM in a tight loop on a box in Asia that sent a few hundred emails per second and saturated the company WAN backbone and mailboxes of the whole team. Good times.

Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

#18

Post by fcarraldo »

reactordev wrote: post #9

Another day, another “Using PostgreSQL for…” thing it wasn’t designed for. This isn’t a good idea. What happens when the queue goes down and all messages are dead lettered? What happens when you end up with competing messages? This is not the way.

There are a ton of job/queue systems out there that are based on SQL DBs. GoodJob and SupaBase Queues are two examples.

It’s not usable for high scale processing but most applications just need a simple queue with low depth and low complexity. If you’re already managing PSQL and don’t want to add more management to your stack (and managed services aren’t an option), this pattern works just fine. Go back 10-15yrs and it was more common, especially in Ruby shops, as teams willing to adopt Kafka/Cassandra/etc were more rare.

Re: Using PostgreSQL as a Dead Letter Queue for Event-Driven Systems

#19

Post by direwolf20 »

reactordev wrote: post #9

Another day, another “Using PostgreSQL for…” thing it wasn’t designed for. This isn’t a good idea. What happens when the queue goes down and all messages are dead lettered? What happens when you end up with competing messages? This is not the way.

The other system you're using that isn't Postgres can also go down.

Many developers overcomplicate systems. In the pursuit of 100% uptime, if you're not extremely careful, you removed more 9s with complexity than you added with redundancy. And although hyperscalers pride themselves on their uptime (Amazon even achieved three nines last year!) in reality most customers of most businesses are fine if your system is down for ten minutes a month. It's not ideal and you should probably fix that, but it's not catastrophic either.

Post reply on HN