Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postgres is a great pub/sub and job server (2019)

webapp.io

Post reply on HN
Print view View original

61–70 of 209 posts

Re: Postgres is a great pub/sub and job server (2019)

#61

Post by daenz »

NavinF wrote: post #59
daenz wrote: post #40

Earlier quoted context omitted.

The transaction feature seems nice but how often is your application dropping queue messages because something happened between tx.commit() and queue.send(msg)? My experience has been that this is not an issue.

Oh that happens fairly often. In fact, some message will be lost every time your queue server reboots due to a power outage, PSU failure, kernel panic, OOM, etc. (Unless it spends almost all of its time idle in which case I guess no messages will be in flight) You’re guaranteed to break the invariant sooner or later so you end up with all the usual complexity of keeping stuff in sync.

Your queue server rebooting is completely orthogonal to whether the application submitting the message can do so atomically or not. Use a cloud service if you care about durability.

Edit>> I see you edited your post after I responded. None of those scenarios qualify as "fairly often."

Re: Postgres is a great pub/sub and job server (2019)

#62

Post by Spivak »

staticassertion wrote: post #32
petilon wrote: post #10

Just because something can be used to do something doesn't mean it should. Kafka is specifically designed for this purpose, it is free, and it is easy to learn and use. If "starting with Postgres and then switching out when the time comes" saves money then I can understand. Otherwise use the right tool for the right job, right from the start.

Kafka is not a queue. Kafka's parallelism is limited by the number of partitions you allocate, and you have to be sure to avoid head of line blocking. Not the case with a queue.

This needs to be sung from the rooftops every time Kafka is mentioned. It's an amazing tool but it is the wrong wrong wrong tool if you need a queue. It will bite you in the ass and you'll be left with someone breathing down your neck wondering why jobs are processing so slowly and why you can't just spin up more workers.

Re: Postgres is a great pub/sub and job server (2019)

#63

Post by daenz »

rockwotj wrote: post #52
colinchartier wrote: post #33

Earlier quoted context omitted.

Such a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!

Fwiw I don't know the shape of the data, but I feel like you could do this with Firebase for a few bucks a month...

you 100% could, and this thread feels like the twilight zone with how many people are advocating for using a rdbms for (what seems like) most peoples queuing needs.

Re: Postgres is a great pub/sub and job server (2019)

#64

Post by ivalm »

dreyfan wrote: post #57
boomskats wrote: post #26

Earlier quoted context omitted.

> doing hundreds of thousands of messages per day > The postgres instance now runs on 32 cores and 128gb of memory and has scaled well. Am I the only one?

It’s such a low throughput requirement I think even bitcoin could support it.

No, this is prob still spiky enough to have more than 7 transactions per second, that’s too much for Bitcoin.

Re: Postgres is a great pub/sub and job server (2019)

#65

Post by tata71 »

daenz wrote: post #63
rockwotj wrote: post #52

Earlier quoted context omitted.

Fwiw I don't know the shape of the data, but I feel like you could do this with Firebase for a few bucks a month...

you 100% could, and this thread feels like the twilight zone with how many people are advocating for using a rdbms for (what seems like) most peoples queuing needs.

Does Firebase offer self-hosting these days?

What do you say to those who don't want Google to know their usage info?

Re: Postgres is a great pub/sub and job server (2019)

#66

Post by lern_too_spel »

colinchartier wrote: post #33
boomskats wrote: post #26

Earlier quoted context omitted.

> doing hundreds of thousands of messages per day > The postgres instance now runs on 32 cores and 128gb of memory and has scaled well. Am I the only one?

Such a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!

It probably fits within the free tier limits of a managed pubsub service.

Re: Postgres is a great pub/sub and job server (2019)

#67

Post by threeseed »

colinchartier wrote: post #33
boomskats wrote: post #26

Earlier quoted context omitted.

> doing hundreds of thousands of messages per day > The postgres instance now runs on 32 cores and 128gb of memory and has scaled well. Am I the only one?

Such a server is 400$/mo, a backend developer that can confidently maintain kafka in production is significantly more expensive!

But Kafka does significantly more.

And if your needs are simpler like in this case then there are dozens of smaller pub/sub/queue systems that you could compare this to.

Re: Postgres is a great pub/sub and job server (2019)

#68

Post by pricci »

tata71 wrote: post #65
daenz wrote: post #63

Earlier quoted context omitted.

you 100% could, and this thread feels like the twilight zone with how many people are advocating for using a rdbms for (what seems like) most peoples queuing needs.

Does Firebase offer self-hosting these days? What do you say to those who don't want Google to know their usage info?

Checkout supabase.com. It is based on postgres.

Re: Postgres is a great pub/sub and job server (2019)

#69

Post by threeseed »

jastr wrote: post #27
petilon wrote: post #10

Just because something can be used to do something doesn't mean it should. Kafka is specifically designed for this purpose, it is free, and it is easy to learn and use. If "starting with Postgres and then switching out when the time comes" saves money then I can understand. Otherwise use the right tool for the right job, right from the start.

This is advice that seems reasonable but is actually pretty harmful. Take a startup with a few users. The senior engineer decides they need pub/sub to ship a new feature. With Kafka, the team goes to learn about Kafka best practices, choose client libraries, and learn the Kafka quirks. They also need to spin up Kafka instances. They ship it in a month. With postgres, they’ve got an MVP in a day, and shipped within a…

How does any of this equally not apply to PostgreSQL ?

Is this some magical database where you don't need to worry about access patterns, best practices or how it is deployed.

Re: Postgres is a great pub/sub and job server (2019)

#70

Post by [deleted] »

daenz wrote: post #61
NavinF wrote: post #59

Earlier quoted context omitted.

Oh that happens fairly often. In fact, some message will be lost every time your queue server reboots due to a power outage, PSU failure, kernel panic, OOM, etc. (Unless it spends almost all of its time idle in which case I guess no messages will be in flight) You’re guaranteed to break the invariant sooner or later so you end up with all the usual complexity of keeping stuff in sync.

Your queue server rebooting is completely orthogonal to whether the application submitting the message can do so atomically or not. Use a cloud service if you care about durability. Edit>> I see you edited your post after I responded. None of those scenarios qualify as "fairly often."

[deleted]
Post reply on HN