Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choose Postgres queue technology

adriano.fyi

Post reply on HN
Print view View original

121–130 of 369 posts

Re: Choose Postgres queue technology

#121

Post by mjb »

bicijay wrote: post #80
vhiremath4 wrote: post #78

We used postgres for some of our queues back when we were at ~10 msg/s. It scaled quite a bit, but, honestly, setting up SQS or some other queue stack in AWS, GCP, or Azure is so simple and purpose built for the task (with DL queues and the like built in), I don’t know why you wouldn’t just go that route and not have to worry about that system shitting the bed and affecting the rest of the DB’s health. It seems fooli…

Transactions, data consistency. This is the answer that you will not find in SQS.

You can't do better than At Least Once if you're having side effects outside the database, so it's not clear that SQS's weaker semantics have any practical effect.

Re: Choose Postgres queue technology

#122

Post by nsonha »

sarchertech wrote: post #106
nsonha wrote: post #76

Earlier quoted context omitted.

Parallel or not, the order is of importance in any queue system.

Unless you can guarantee that the processing time of each job is exactly the same, if you have multiple workers processing the same queue, you can’t order anything except the start time. You can use locks to effectively break the queue into sub queues so that each sub queue is only being processed by 1 worker. Then you can order that sub queue.

job should be attempted inthe same order/priority they are enqueued, that's the meaning of the word "queue". That they take varrying amounts of time is another matter.

Re: Choose Postgres queue technology

#123

Post by lmm »

mianos wrote: post #91

Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. Having transactions is quite handy. https://wiki.postgresql.org/wiki/SkyTools I did a few talks on this at Sydpy as I used it at work quite a bit. It's…

> Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern.

It works great until it doesn't, and the way it breaks puts you in a state that's very difficult to recover from. And if your excuse for using a database as a queue was that you were already running a database, that cuts both ways: congratulations, your queue mess has now brought down your primary datastore too.

Re: Choose Postgres queue technology

#124

Post by LennyWhiteJr »

klysm wrote: post #61
orangepanda wrote: post #23

Earlier quoted context omitted.

As I understand, with SKIP LOCKED rows would no longer be processed in-order?

Depends on how many consumers you have. If you need order guarantees, then something like the outbox pattern is probably a better fit.

Nothing about the outbox pattern guarantees ordering.

Re: Choose Postgres queue technology

#126

Post by LennyWhiteJr »

In my Amazon team, we use PostgreSQL as a queue using skip-locked to implement transactional outbox pattern for our database inserts. People commenting 'just use a queue' are totally missing the need for transactional consistency. I agree with the author, it's an amazing tool and scales quite well.

Re: Choose Postgres queue technology

#128

Post by jerrygenser »

liminal wrote: post #99

My main issue with pretty much all queue approaches is that they don't work across platforms. They are built for one technology stack, be it Python/NodeJS/etc. This is fine if you've only got one stack, but in a microservices world it doesn't work where jobs can span multiple systems. You might be able to find some abandoned library that supports that queue tech on the other platforms you need, but now you've basical…

There are queues like sqs and google pubsub where there is already official clients written for most popular languages. And they also have http API so you can use it in any language environment that can make http requests.

Re: Choose Postgres queue technology

#129

Post by rockwotj »

All of shortwave.com is built on this concept. The super powerful bit here that is the mentioned is that enqueuing jobs is transactional with other normal transactions, so you don’t have to architect around using a different system for the queue and the rest of your data.

Re: Choose Postgres queue technology

#130

Post by dventimi »

lmm wrote: post #123
mianos wrote: post #91

Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. Having transactions is quite handy. https://wiki.postgresql.org/wiki/SkyTools I did a few talks on this at Sydpy as I used it at work quite a bit. It's…

> Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. It works great until it doesn't, and the way it breaks puts you in a state that's very difficult to recover from. And if your excuse for using a data…

I wonder if it would be possible to have more than just one PostgreSQL database.
Post reply on HN