Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choose Postgres queue technology

adriano.fyi

Post reply on HN
Print view View original

131–140 of 369 posts

Re: Choose Postgres queue technology

#131

Post by hn_throwaway_99 »

lmm wrote: post #123
mianos wrote: post #91

Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. Having transactions is quite handy. https://wiki.postgresql.org/wiki/SkyTools I did a few talks on this at Sydpy as I used it at work quite a bit. It's…

> Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. It works great until it doesn't, and the way it breaks puts you in a state that's very difficult to recover from. And if your excuse for using a data…

> congratulations, your queue mess has now brought down your primary datastore too.

Just don't put your queue tables/logic in the same DB instance as your datastore. There are still a lot of benefits to using the same tech even if you have segregated instances.

Re: Choose Postgres queue technology

#132

Post by mianos »

reubenbond wrote: post #113
mianos wrote: post #91

Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. Having transactions is quite handy. https://wiki.postgresql.org/wiki/SkyTools I did a few talks on this at Sydpy as I used it at work quite a bit. It's…

> Anything but RabbitMQ. Would you mind elaborating on this? I'd be happy for others to chime in with their experiences/opinions, too.

The software works excellently in a development environment and performs well when running as a single instance. However, I encountered issues when scaling it up for high availability in a clustered setup. The system would fail inconsistently, with two masters consuming messages simultaneously, which wasn't ideal for my use case. Eventually, I switched to Kafka and haven't revisited the original solution since.

It's worth noting that these issues might have been due to my improper configuration. Nevertheless, if the configuration process is fraught with pitfalls, that's problematic in itself. I've had these experiences more than once.

Additionally, I found a critical race condition in the Python library, rendering it practically unusable for me. I submitted a bug report with a minimal example demonstrating the issue. I considered fixing it myself, but since using RabbitMQ wasn't crucial for my project, I switched to ZeroMQ, which didn't require a broker. The issue was acknowledged and fixed about a year later. At the time, I had to assume that nobody else was using the Python bindings.

Three years ago, I worked on a project that used the software for a Celery queue. Messages would occasionally go missing, although this could have been a configuration issue on our part. Ultimately, we replaced it with a Redis queue (not the best practice, I admit) and didn't look back. This was for a lower-availability use case where a single instance of Redis sufficed.

Re: Choose Postgres queue technology

#133

Post by hn_throwaway_99 »

mianos wrote: post #91

Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. Having transactions is quite handy. https://wiki.postgresql.org/wiki/SkyTools I did a few talks on this at Sydpy as I used it at work quite a bit. It's…

> This said, I'd use a dedicated queue these days.

I agree, primary reason being that if you're in the cloud (thus this applies to a lot of people but obviously not everyone), all the cloud providers have extremely easy to use, and cheap, hosted queueing tech. Even if you're worried about vendor lockin, queueing primitives are so small (basically push and pop), that it's relatively easy to write things in a way so it would be easy to migrate if necessary.

Re: Choose Postgres queue technology

#134

Post by lmm »

dventimi wrote: post #130
lmm wrote: post #123

Earlier quoted context omitted.

> Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. It works great until it doesn't, and the way it breaks puts you in a state that's very difficult to recover from. And if your excuse for using a data…

I wonder if it would be possible to have more than just one PostgreSQL database.

It is, but it comes with a lot of the same costs as having a PostgreSQL database and a proper queue system.

Re: Choose Postgres queue technology

#135

Post by mianos »

lmm wrote: post #123
mianos wrote: post #91

Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. Having transactions is quite handy. https://wiki.postgresql.org/wiki/SkyTools I did a few talks on this at Sydpy as I used it at work quite a bit. It's…

> Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. It works great until it doesn't, and the way it breaks puts you in a state that's very difficult to recover from. And if your excuse for using a data…

> It works great until it doesn't, and the way it breaks puts you in a state that's very difficult to recover from.

This is similar to saying, 'if I mess up all the tables in one database I wreck the rest'. Just my opinion, but this is not actually a thing in databases.

Maybe compromised the performance of one database due to another loading things up? I think database are developed with this as an important consideration. I can't say I have seen this, but admit it's a possibility.

Also, if you run one postgres, you won't have of an issue running another if you have the experience in production.

Re: Choose Postgres queue technology

#137

Post by mianos »

lmm wrote: post #134
dventimi wrote: post #130

Earlier quoted context omitted.

I wonder if it would be possible to have more than just one PostgreSQL database.

It is, but it comes with a lot of the same costs as having a PostgreSQL database and a proper queue system.

No, you already know how to run and manager a postgres database.

If have you large teams, for sure, pick a proper queue that someone in the team knows well.

Re: Choose Postgres queue technology

#138

Post by lmm »

mianos wrote: post #137
lmm wrote: post #134

Earlier quoted context omitted.

It is, but it comes with a lot of the same costs as having a PostgreSQL database and a proper queue system.

No, you already know how to run and manager a postgres database. If have you large teams, for sure, pick a proper queue that someone in the team knows well.

> No, you already know how to run and manager a postgres database.

Well, maybe. Postgres is large and has lots of features. NOTIFY/LISTEN is a distinct thing with its own quirks, and just because you've been running a postgres database up until now doesn't mean you're going to know about the admin side of these other features.

Re: Choose Postgres queue technology

#139

Post by teaearlgraycold »

jpgvm wrote: post #28

Few things. 1. The main downside to using PostgreSQL as a pub/sub bus with LISTEN/NOTIFY is that LISTEN is a session feature, making it incompatible with statement level connection pooling. 2. If you are going to do this use advisory locks [0]. Other forms of explicit locking put more pressure on the database while advisory locks are deliberately very lightweight. My favorite example implementation is que [1] which i…

Is there a Node version of Que?

Re: Choose Postgres queue technology

#140

Post by lmm »

mianos wrote: post #135
lmm wrote: post #123

Earlier quoted context omitted.

> Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. It works great until it doesn't, and the way it breaks puts you in a state that's very difficult to recover from. And if your excuse for using a data…

> It works great until it doesn't, and the way it breaks puts you in a state that's very difficult to recover from. This is similar to saying, 'if I mess up all the tables in one database I wreck the rest'. Just my opinion, but this is not actually a thing in databases. Maybe compromised the performance of one database due to another loading things up? I think database are developed with this as an important consider…

> This is similar to saying, 'if I mess up all the tables in one database I wreck the rest'. Just my opinion, but this is not actually a thing in databases.

If you mess up the tables in one database it doesn't affect others, but if you lock up the server where it can't respond to queries, that affects every database running on that server.

> Also, if you run one postgres, you won't have of an issue running another if you have the experience in production.

We're talking about using a different feature that you presumably haven't used before, so you won't necessarily know about the admin side of that.

Post reply on HN