Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres: A better message queue than Kafka?

dagster.io

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Postgres: A better message queue than Kafka?

#31

Post by manv1 »

eurasiantiger wrote: post #7
mannyv wrote: post #4

You can spew messages into sqs then have a lambda on sqs sending to pg. Funny, we were going to use kafka too, but just sending to mysql worked just fine . One day that will change, obviously.

It works fine on the happy path, but if mysql goes down, you lose things. Having a distributed, resilient queue has availability benefits.

That's why we have an RDS cluster. We're not multi-region because, frankly, if the region barfs we can blame it on Amazon. And in any case we'll still deliver just fine as long as cloudfront keeps going.

Re: Postgres: A better message queue than Kafka?

#32

Post by manv1 »

eurasiantiger wrote: post #7
mannyv wrote: post #4

You can spew messages into sqs then have a lambda on sqs sending to pg. Funny, we were going to use kafka too, but just sending to mysql worked just fine . One day that will change, obviously.

It works fine on the happy path, but if mysql goes down, you lose things. Having a distributed, resilient queue has availability benefits.

Also, we use SQS as an intermediary between our logging and mysql, so even if mysql goes down SQS will keep things for...a while.

Re: Postgres: A better message queue than Kafka?

#34

Post by andrewstuart »

hbrn wrote: post #20
andrewstuart wrote: post #17

Not just Postgres. You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED. Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible. You could write a file system based message queue i…

What makes it atomic is running publishers and consumers on the same box (since you're sharing filesystem between those). Also listdir is a big bottleneck here: while True: # get files in outbox files_in_outbox = [f'{PREFIX}/outbox/{x}' for x in os.listdir(f'{PREFIX}/outbox')]

>> What makes it atomic is running publishers and consumers on the same box (since you're sharing filesystem between those).

It's the move/rename that is atomic.

https://man7.org/linux/man-pages/man2/rename.2.html

Re: Postgres: A better message queue than Kafka?

#35

Post by hbrn »

andrewstuart wrote: post #34
hbrn wrote: post #20

Earlier quoted context omitted.

What makes it atomic is running publishers and consumers on the same box (since you're sharing filesystem between those). Also listdir is a big bottleneck here: while True: # get files in outbox files_in_outbox = [f'{PREFIX}/outbox/{x}' for x in os.listdir(f'{PREFIX}/outbox')]

>> What makes it atomic is running publishers and consumers on the same box (since you're sharing filesystem between those). It's the move/rename that is atomic. https://man7.org/linux/man-pages/man2/rename.2.html

Not really.

       However, there will
       probably be a window in which both oldpath and newpath refer to
       the file being renamed.
But that's not even the main point.

1. Move happens after email is sent, so there is a window where email is already being sent but file still exists. 2. Even if you do it before, there's still a window between os.listdir() and os.remove() 3. Complexity is O(N^2) due to listdir() + getctime() being called on every iteration.

If you just want to ensure order, it probably works fine at a small scale. But it would be unwise to run multiple consumers on a single instance, and impossible to run them on multiple instances.

Re: Postgres: A better message queue than Kafka?

#36

Post by EdwardDiego »

pritambaral wrote: post #13
eurasiantiger wrote: post #7

Earlier quoted context omitted.

It works fine on the happy path, but if mysql goes down, you lose things. Having a distributed, resilient queue has availability benefits.

> It works fine on the happy path, but if mysql goes down, you lose things. Why do you have to lose things if the DB goes down? Agreed, untuned & unconfigured MySQL (and MongoDB) out-of-the-box can lose things due to bugs and design issues, but that is the case even when they are running. However, DBs, in general are made precisely for the purpose of safely storing things and not losing them. OTOH, the number of Kafk…

> OTOH, the number of Kafka setups I have seen that'd lose things when something goes down ...

Was the data actually acked across all replicas by Kafka? If so, very interested in how it was lost.

Re: Postgres: A better message queue than Kafka?

#38

Post by EdwardDiego »

hbrn wrote: post #18
hbrn wrote: post #9

A controversial, but a very pragmatic take. Queues are great for semi-infinite scalability, but you rarely need it. There's numerous subtle benefits to using db compared to regular message queues that are often overlooked. Being able to delete, reorder, or edit specific messages can be a lifesaver when things go wrong. Most common issue with queue-based systems is getting overwhelmed with messages. Either your consum…

Here's just a few examples from my experience: 1. Huge number of messages from test system were accidentally inserted into production. Queue solution: disable consumers, move messages to a temporary queue while filtering them, move messages back to the old queue, enable consumers DB solution: just delete rogue messages 2. We want to store some of the messages, but we're not ready to process them yet Queue solution: c…

1. Kafka isn't a message queue, but advance consumer offsets past bad data. Hit person who inserted test data into prod with a cricket bat and then hit person who designed a system where they could do that with the same cricket bat, but harder.

2. Kafka - do what you did with your DB consumers.

3. Kafka - consumers going down can't cause duplicates, what's even going on with your queue?

Post reply on HN