Live data from Hacker News

Postgres: A better message queue than Kafka?

dagster.io

41–47 of 47 posts

Re: Postgres: A better message queue than Kafka?

#41

Not just Postgres. You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED. Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible. You could write a file system based message queue i…

love to see this kind of hackery

I love to see effective yet brutally-simple "redneck engineering" solutions to software problems, particularly ones that straddle the line between genius and stupidity in a way that makes architecture astronauts feel uncomfortable.

I used to work at an brokerage that worked with a panel of around 12 providers, all of whom offered 5+ products that were updated multiple times per year to meet changing regulatory requirements. When a sales adviser recommended product X from provider Y to a customer, the adviser would then need to fill in an application form that was precisely tailored to that particular revision of that particular product. Bear in mind that these were complex, multi-sectioned forms. Needless to say, this created a huge workload for the devteam to keep track of all the product changes and update the UI accordingly each time.

At some point, someone on the devteam had the genius idea to simply take the PDF application forms from the provider, extract the individual pages as PNGs and overlay HTML form elements on top of them. The provider would essentially be doing our UI design for us. Add in an editor tool so the sales managers could set up the forms themselves and a tagging system so specific fields could be pre-filled from customer data we already had stored in the DB and the devteam's workload dropped by maybe 90%. Simple, stupid perhaps, but effective.

Re: Postgres: A better message queue than Kafka?

#42
“ Today these are not major problems for us, and frankly, I’m not sure how we’ll tackle these problems. Maybe we’ll grow multi-region support and buy really big Postgres boxes. Maybe we’ll move to sharded Postgres. Maybe we’ll switch to Kafka or some other distributed queue. ”

LOL

Re: Postgres: A better message queue than Kafka?

#43
post #18

Earlier quoted context omitted.

Here's just a few examples from my experience: 1. Huge number of messages from test system were accidentally inserted into production. Queue solution: disable consumers, move messages to a temporary queue while filtering them, move messages back to the old queue, enable consumers DB solution: just delete rogue messages 2. We want to store some of the messages, but we're not ready to process them yet Queue solution: c…

1. Kafka isn't a message queue, but advance consumer offsets past bad data. Hit person who inserted test data into prod with a cricket bat and then hit person who designed a system where they could do that with the same cricket bat, but harder. 2. Kafka - do what you did with your DB consumers. 3. Kafka - consumers going down can't cause duplicates, what's even going on with your queue?

> what's even going on with your queue?

A message with the latest state of an object gets pushed into queue. When noone is consuming those, you essentially get duplicates: queue contains all of the updates, but you only care about the most recent one.

Re: Postgres: A better message queue than Kafka?

#44
post #35

Earlier quoted context omitted.

>> What makes it atomic is running publishers and consumers on the same box (since you're sharing filesystem between those). It's the move/rename that is atomic. https://man7.org/linux/man-pages/man2/rename.2.html

Not really. However, there will probably be a window in which both oldpath and newpath refer to the file being renamed. But that's not even the main point. 1. Move happens after email is sent, so there is a window where email is already being sent but file still exists. 2. Even if you do it before, there's still a window between os.listdir() and os.remove() 3. Complexity is O(N^2) due to listdir() + getctime() being…

I worked with a network filesystem that supported atomic renames and we based an entire large-scale production system on the idea that it would work (it did). The system supported Youtube and Google Play model training, regularly processing increments of hundreds of terabytes.

Re: Postgres: A better message queue than Kafka?

#45

Not just Postgres. You can do exactly this with MySQL and SQL server too because they both support SKIP LOCKED. Interestingly, the plain old file system on Linux also makes the basis of a perfectly acceptable message queue for many use cases - the thing that makes it work is that the file move operation is atomic. Atomic moves are what make queuing systems possible. You could write a file system based message queue i…

We found difficulty with the purely advisory locking that Linux had. Also the possibility of network filesystems made it a pain. Do you have any experience with either?

I also found that advisory locking has a lot of gotchas, especially when used in multithread contexts (apparently you can lose the lock because a different thread closed a different file descriptor on the same file).

Re: Postgres: A better message queue than Kafka?

#47
post #5

"The write path. We built a daemon that would select log entries that were older than two weeks, copy them into a file in S3, and delete the rows from the database" Seriously... How can you ever consider saying "a rdbms is just fine as a kafka alternative" under those conditions ?

Why not?

Because that would be equivalent to saying « rdbms are fine as a kafka alternative provided you build a compensation mechanism for all the things it does worse than kafka». It doesn’t make any sense.
Post reply on HN