Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres is a great pub/sub and job server (2019)

webapp.io

Post reply on HN
Print view View original

21–30 of 209 posts

Re: Postgres is a great pub/sub and job server (2019)

#21

Post by staticassertion »

daenz wrote: post #6

Strong disagree on using a database as a message queue. This article[0] covers many of the reasons why. Summary: additional application complexity and doesn't scale well with workers. 0. https://www.cloudamqp.com/blog/why-is-a-database-not-the-rig... EDIT>> I am not suggesting people build their own rabbitmq infrastructure. Use a cloud service. The article is informational only.

It seems notable that this is a blog post from the perspective of rabbitmq authors, or at least the author of a book about it. It talks very vaguely about one potential implementation of a queue on postgres.

Re: Postgres is a great pub/sub and job server (2019)

#22

Post by anotherhue »

petilon wrote: post #10

Just because something can be used to do something doesn't mean it should. Kafka is specifically designed for this purpose, it is free, and it is easy to learn and use. If "starting with Postgres and then switching out when the time comes" saves money then I can understand. Otherwise use the right tool for the right job, right from the start.

I've been on stage at KafkaConf demo-ing my Kafka SRE chops, and I would avoid Kafka until I am sure it is necessary.

'easy to learn and use' is a downright lie.

edit: link to this same topic being discussed a few weeks ago: https://news.ycombinator.com/item?id=28903614#28904103

Re: Postgres is a great pub/sub and job server (2019)

#24

Post by anonu »

colinchartier wrote: post #19

Author here! A few updates since this was published two years ago: - The service mentioned (now called https://webapp.io ) eventually made it into YC (S20) and still uses postgres as its pub/sub implementation, doing hundreds of thousands of messages per day. The postgres instance now runs on 32 cores and 128gb of memory and has scaled well. - We bolstered Postgres's PUBLISH with Redis pub/sub for high traffic code p…

Thanks for the great blog post - still relevant after a few years!

> statement_timeout=(a few days)

wouldnt you want this to be a few seconds or minutes? Maybe I miss the point of setting this to days...

Re: Postgres is a great pub/sub and job server (2019)

#26

Post by boomskats »

colinchartier wrote: post #19

Author here! A few updates since this was published two years ago: - The service mentioned (now called https://webapp.io ) eventually made it into YC (S20) and still uses postgres as its pub/sub implementation, doing hundreds of thousands of messages per day. The postgres instance now runs on 32 cores and 128gb of memory and has scaled well. - We bolstered Postgres's PUBLISH with Redis pub/sub for high traffic code p…

> doing hundreds of thousands of messages per day

> The postgres instance now runs on 32 cores and 128gb of memory and has scaled well.

Am I the only one?

Re: Postgres is a great pub/sub and job server (2019)

#27

Post by jastr »

petilon wrote: post #10

Just because something can be used to do something doesn't mean it should. Kafka is specifically designed for this purpose, it is free, and it is easy to learn and use. If "starting with Postgres and then switching out when the time comes" saves money then I can understand. Otherwise use the right tool for the right job, right from the start.

This is advice that seems reasonable but is actually pretty harmful.

Take a startup with a few users. The senior engineer decides they need pub/sub to ship a new feature. With Kafka, the team goes to learn about Kafka best practices, choose client libraries, and learn the Kafka quirks. They also need to spin up Kafka instances. They ship it in a month.

With postgres, they’ve got an MVP in a day, and shipped within a week.

Re: Postgres is a great pub/sub and job server (2019)

#28

Post by colinchartier »

anonu wrote: post #24
colinchartier wrote: post #19

Author here! A few updates since this was published two years ago: - The service mentioned (now called https://webapp.io ) eventually made it into YC (S20) and still uses postgres as its pub/sub implementation, doing hundreds of thousands of messages per day. The postgres instance now runs on 32 cores and 128gb of memory and has scaled well. - We bolstered Postgres's PUBLISH with Redis pub/sub for high traffic code p…

Thanks for the great blog post - still relevant after a few years! > statement_timeout=(a few days) wouldnt you want this to be a few seconds or minutes? Maybe I miss the point of setting this to days...

Didn't want to deal with ramifications of statement timeouts in a complex system, the failure mode mentioned (queue filling up) happened on the scale of 6 weeks, so it was very cheap operationally to set this timeout to some high value.

Re: Postgres is a great pub/sub and job server (2019)

#29

Post by peterhunt »

paxys wrote: post #13
kasey_junk wrote: post #3

Largely agree at the scale this article is working with. But frankly, if 10k/s inserts is the scale you are talking about even worrying about a pub/sub solution seems odd. Introducing something like Kafka for anything less than an order of magnitude more than that seems like an architectural blunder. By the time you are there Postgres will have obviously disqualified itself.

What does scale have to do with it? Pub/sub as an architectural pattern could be equally relevant for your use case whether there are a hundred users in your system or a billion. And Kafka isn't the only solution for it. There are many lightweight pub/sub and queuing systems which also don't involve needlessly adding abstraction layers and application code into an RDBMS.

I think massive scale is the only reason you'd really want to adopt something like Kafka. If you're 10k inserts/s or less then there's no reason not to do everything in a single big relational DB where you get the warm fuzzy feeling of transactions, point in time backups, scalable read replication, etc

Re: Postgres is a great pub/sub and job server (2019)

#30

Post by seanp2k2 »

vlmutolo wrote: post #8

I had thought about using postgres as a job queue before, but I couldn't figure out in my head how to make sure two processes didn't both take the same job. The "FOR UPDATE" and "SKIP LOCKED" were the keys to make this work in the article. Essentially, as far as I can tell, "SELECT FOR UPDATE" locks the rows as they're selected (locks are apparently visible outside the transaction), and "SKIP LOCKED" skips over rows…

This. The article seems like "one weird trick that message queue companies HATE" as it's utilizing, as far as I understand, some SQL semantics in a very specific way to cobble together a way of achieving what other software is designed to do out of the box. It seems fine for a toy system, but I wouldn't stake the success of a real company on this approach.

One could also use DNS TXT as an RDBMS with some interesting fault tolerance and distribution schemes. That doesn't mean it's a good idea or the best way to solve a problem.

If you haven't seen them already, the Jepsen analyses are really worth a read: https://aphyr.com/posts/293-jepsen-kafka https://aphyr.com/posts/282-jepsen-postgres https://aphyr.com/tags/jepsen

Post reply on HN