Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres: A better message queue than Kafka?

dagster.io

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Postgres: A better message queue than Kafka?

#21

Post by xupybd »

lightbendover wrote: post #8

File this under "If a headline is asking a question, then the answer is NO." Honestly, I'm not even sure what point the author is trying to make besides "anecdotally and at low scale, unusual-for-the-purpose technology X solved problem Y." A near-infinite number of bad patterns can solve problems along happy paths and resolve plenty of edge cases to boot. 99.9% availability was a goal post here? There are systems whe…

This solves their problems not some theoretical other problem.

They avoided over engineering and building more complexity than required.

Re: Postgres: A better message queue than Kafka?

#22

Post by xupybd »

bsaul wrote: post #5

"The write path. We built a daemon that would select log entries that were older than two weeks, copy them into a file in S3, and delete the rows from the database" Seriously... How can you ever consider saying "a rdbms is just fine as a kafka alternative" under those conditions ?

How does Kafka solve this problem any better?

Your messages expire so if you want to archive you need a consumer to write them to disk. That sounds very similar.

Re: Postgres: A better message queue than Kafka?

#23

Post by chrsig »

Something I think needs to be stated: Kafka is not a message queue. If you need things like work stealing, you're going to have a bad time. It doesn't seem to apply in this case, but the terminology misuse is going to lead some poor developer astray.

It seems like they only tried postgres? so they didn't compare it with kafka at all....so yeah, you can make a message queue or log table in a rdbms.

Re: Postgres: A better message queue than Kafka?

#24

Post by smeagull »

andrewstuart wrote: post #17

Not just Postgres. You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED. Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible. You could write a file system based message queue i…

We found difficulty with the purely advisory locking that Linux had. Also the possibility of network filesystems made it a pain.

Do you have any experience with either?

Re: Postgres: A better message queue than Kafka?

#25

Post by andrewstuart »

smeagull wrote: post #24
andrewstuart wrote: post #17

Not just Postgres. You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED. Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible. You could write a file system based message queue i…

We found difficulty with the purely advisory locking that Linux had. Also the possibility of network filesystems made it a pain. Do you have any experience with either?

You don't lock, you move the file that it next to be processed. File moves are atomic. You move the file out of the list of files that are being picked up for processing.

Lock free.

Re: Postgres: A better message queue than Kafka?

#26

Post by whalesalad »

andrewstuart wrote: post #17

Not just Postgres. You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED. Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible. You could write a file system based message queue i…

love to see this kind of hackery

Re: Postgres: A better message queue than Kafka?

#27

Post by whalesalad »

hbrn wrote: post #9

A controversial, but a very pragmatic take. Queues are great for semi-infinite scalability, but you rarely need it. There's numerous subtle benefits to using db compared to regular message queues that are often overlooked. Being able to delete, reorder, or edit specific messages can be a lifesaver when things go wrong. Most common issue with queue-based systems is getting overwhelmed with messages. Either your consum…

Throughput is often where an RDBMS falls over here. Unless you have partitions/isolation around your high throughput workloads (or throw tons of horsepower/$$$ at it) you can’t achieve the same ops/sec as a dedicated queueing mechanism.

But you do have a lot of valid points, particularly about failure recovery. My approach tends to involve a hybrid of the two if high speed is a necessary component somewhere. The system can restore its runtime state from a db, but the active mechanics of it are running on message passing of some kind.

Re: Postgres: A better message queue than Kafka?

#28

Post by andrewstuart »

smeagull wrote: post #24
andrewstuart wrote: post #17

Not just Postgres. You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED. Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible. You could write a file system based message queue i…

We found difficulty with the purely advisory locking that Linux had. Also the possibility of network filesystems made it a pain. Do you have any experience with either?

Network file systems do not support atomic moves, but you should not run such an application on a network file system.

Re: Postgres: A better message queue than Kafka?

#29

Post by [deleted] »

lightbendover wrote: post #14
hbrn wrote: post #12

Earlier quoted context omitted.

Nothing is more profitable than serving ads. I'm still waiting for an answer.

While I have no idea what kind of distributed system would need 7+ 9s, there are plenty of solution providers who at least promise it as part of their marketing. I truly disbelieve that these companies don't have any point of failure within their system that would take them under that threshold, even if it hasn't been challenged to date, but that level of availability is still the published goal. Hopefully needless t…

[deleted]

Re: Postgres: A better message queue than Kafka?

#30

Post by hk1337 »

I don’t want to doubt Postgres’ effectiveness as a message queue (however difficult it may be) but I have to wonder if maybe they were doing something weird or wrong in the consumer for it to be ineffective?

Maybe it’s not that Postgres is a better than Kafka as much as Postgres is a better solution for what they were trying to do and use Kafka?

Post reply on HN