Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres: A better message queue than Kafka?

dagster.io

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Postgres: A better message queue than Kafka?

#11

Post by lightbendover »

hbrn wrote: post #10
lightbendover wrote: post #8

File this under "If a headline is asking a question, then the answer is NO." Honestly, I'm not even sure what point the author is trying to make besides "anecdotally and at low scale, unusual-for-the-purpose technology X solved problem Y." A near-infinite number of bad patterns can solve problems along happy paths and resolve plenty of edge cases to boot. 99.9% availability was a goal post here? There are systems whe…

What are those mythical systems where seven nines is unacceptable? What are the chances you're going to work on one of those? Google Spanner for example is up to five nines. Are there a lot of systems that need to be three orders of magnitude more reliable than Google Ads?

You're right, nothing is more important than serving ads.

Re: Postgres: A better message queue than Kafka?

#12

Post by hbrn »

lightbendover wrote: post #11
hbrn wrote: post #10

Earlier quoted context omitted.

What are those mythical systems where seven nines is unacceptable? What are the chances you're going to work on one of those? Google Spanner for example is up to five nines. Are there a lot of systems that need to be three orders of magnitude more reliable than Google Ads?

You're right, nothing is more important than serving ads.

Nothing is more profitable than serving ads.

I'm still waiting for an answer.

Re: Postgres: A better message queue than Kafka?

#13

Post by pritambaral »

eurasiantiger wrote: post #7
mannyv wrote: post #4

You can spew messages into sqs then have a lambda on sqs sending to pg. Funny, we were going to use kafka too, but just sending to mysql worked just fine . One day that will change, obviously.

It works fine on the happy path, but if mysql goes down, you lose things. Having a distributed, resilient queue has availability benefits.

> It works fine on the happy path, but if mysql goes down, you lose things.

Why do you have to lose things if the DB goes down? Agreed, untuned & unconfigured MySQL (and MongoDB) out-of-the-box can lose things due to bugs and design issues, but that is the case even when they are running. However, DBs, in general are made precisely for the purpose of safely storing things and not losing them.

OTOH, the number of Kafka setups I have seen that'd lose things when something goes down ... maybe this is not a guaranteed win for the Kafka side of arguments.

> Having a distributed, resilient queue has availability benefits.

High availability is not a function exclusive to Kafka. On the other hand, there's some functions that may come in handy to use in a queue that Kafka simply cannot provide, but DBs can. Off the top of my head: ACID, instant scalability (both up and down) of consumer groups, and the sheer flexibility (and power) that comes with a DB in general.

----

Overall, there's some merits to using a distributed log as a message queue, sure, but there are also merits to using a DB for that.

Re: Postgres: A better message queue than Kafka?

#14

Post by lightbendover »

hbrn wrote: post #12
lightbendover wrote: post #11

Earlier quoted context omitted.

You're right, nothing is more important than serving ads.

Nothing is more profitable than serving ads. I'm still waiting for an answer.

While I have no idea what kind of distributed system would need 7+ 9s, there are plenty of solution providers who at least promise it as part of their marketing. I truly disbelieve that these companies don't have any point of failure within their system that would take them under that threshold, even if it hasn't been challenged to date, but that level of availability is still the published goal. Hopefully needless to say, but 99.9% is not acceptable for most "important" (and yes, advertising is important) applications outside of US-East-1 somehow. Quick examples, no experience with any of these: * https://www.infinidat.com/en/news/press-releases/infinidat-h... * https://vindicia.com/blog/9999999-global-system-uptime/ * https://www.ibm.com/downloads/cas/A856LOWK

Bad examples? Certainly yes. Are there any good examples? Maybe, doubtful. Does any system actually need 7 9s? Maybe, doubtful.

Re: Postgres: A better message queue than Kafka?

#15

Post by peterhunt »

bsaul wrote: post #5

"The write path. We built a daemon that would select log entries that were older than two weeks, copy them into a file in S3, and delete the rows from the database" Seriously... How can you ever consider saying "a rdbms is just fine as a kafka alternative" under those conditions ?

Why not?

Re: Postgres: A better message queue than Kafka?

#16

Post by greymalik »

srajabi wrote: post #6

I don’t think this article made a compelling reason not to use Kafka. In fact it may have made the opposite point. Wouldn’t it have been easier to just use Kafka?

From TFA Postgres was an established tool in their org they already had expertise with. The author argues that adopting a new tool would have been risky and costly.

Re: Postgres: A better message queue than Kafka?

#17

Post by andrewstuart »

Not just Postgres.

You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED.

Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible.

You could write a file system based message queue in 100 lines of async python, which I did here:

https://github.com/bootrino/arniesmtpbufferserver

File system based message queues can be written in any language, extremely simple and, most importantly - zero configuration. One of the most frustrating things about queuing systems is configuration - that includes database backed queuing systems. They can also be fast - I wrote one in Rust which maxed out the hard disk's random write capability well before maxing out the CPU - from memory it beat most of the common queuing systems in terms of messages per second.

Not all use cases for queues need to be able to globally distributed messages queues with the sort of guarantees needed for financial transaction processing. I would suggest to you that in fact most queues out there are used as outbound SMTP queues, which are then over engineered to use something like Celery, which is a nightmare to configure and debug.

Re: Postgres: A better message queue than Kafka?

#18

Post by hbrn »

hbrn wrote: post #9

A controversial, but a very pragmatic take. Queues are great for semi-infinite scalability, but you rarely need it. There's numerous subtle benefits to using db compared to regular message queues that are often overlooked. Being able to delete, reorder, or edit specific messages can be a lifesaver when things go wrong. Most common issue with queue-based systems is getting overwhelmed with messages. Either your consum…

Here's just a few examples from my experience:

1. Huge number of messages from test system were accidentally inserted into production.

Queue solution: disable consumers, move messages to a temporary queue while filtering them, move messages back to the old queue, enable consumers

DB solution: just delete rogue messages

2. We want to store some of the messages, but we're not ready to process them yet

Queue solution: create a separate queue for each message type, insert messages to different queues, manually move them when you're ready to process them (and keep in mind that not every queue can persist messages forever, SQS for example can't hold messages longer than 14 days)

DB solution: just skip those messages while processing

3. Consumers went down and the queue now contains a big number of duplicate messages. While it was fine to just wait a couple hours to let it stabilize, a whale customer started complaining

Queue solution: none (any hacky solution would take longer than it takes for the system to naturally stabilize)

DB solution: move whale customer messages to the front of the queue

Re: Postgres: A better message queue than Kafka?

#19

Post by andrewstuart »

peterhunt wrote: post #15
bsaul wrote: post #5

"The write path. We built a daemon that would select log entries that were older than two weeks, copy them into a file in S3, and delete the rows from the database" Seriously... How can you ever consider saying "a rdbms is just fine as a kafka alternative" under those conditions ?

Why not?

Ideology.

There's alot of people who are ideologically opposed to database backed message queues. They're usually reluctant to give detailed explanations why, because it's an emotional thing.

Re: Postgres: A better message queue than Kafka?

#20

Post by hbrn »

andrewstuart wrote: post #17

Not just Postgres. You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED. Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible. You could write a file system based message queue i…

What makes it atomic is running publishers and consumers on the same box (since you're sharing filesystem between those).

Also listdir is a big bottleneck here:

    while True:
        # get files in outbox
        files_in_outbox = [f'{PREFIX}/outbox/{x}' for x in os.listdir(f'{PREFIX}/outbox')]
Post reply on HN