Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choose Postgres queue technology

adriano.fyi

Post reply on HN
Print view View original

161–170 of 369 posts

Re: Choose Postgres queue technology

#161

Post by pqdbr »

phamilton wrote: post #151
reubenbond wrote: post #113

Earlier quoted context omitted.

> Anything but RabbitMQ. Would you mind elaborating on this? I'd be happy for others to chime in with their experiences/opinions, too.

I can share our experience with RabbitMQ/SQS/Sidekiq. Our two major issues have been around the retry mechanism and resource bottlenecks. The key retry problem is "What happens when a worker crashes?". RabbitMQ solves this problem by tying "unacknowledged messages" to a tcp connection. If the connection dies, the in-flight messages are made available to other connections. This is a decent approach, but we hit a lot o…

> Sidekiq solves "What happens when a worker crashes?" by just not solving it. In the free version, those jobs are just lost.

I've been using Sidekiq for 11+ years in production and I've never seen this happen. Sidekiq (free version) has a very robust retry workflow. What are you talking about here?

Re: Choose Postgres queue technology

#162

Post by Rapzid »

Not sure this particular article brings anything new to the table, but it's nice to spread the word.

Been meaning to build an Orleans stream provider for Postgres.. I believe that's the main missing component that would allow everything to "JustWork" with Postgres until you outgrow it.

Re: Choose Postgres queue technology

#163

Post by AlexITC »

I have been involved in a few projects using postgres-bakend queues for a few years, scale hasn't been a problem so far.

On the other hand, I have done a few experiments with postgres LISTEN/NOTIFY, while the feature seems nice at first glance, I concluded that it wasn't worth it for our use cases, maybe it is different in other languages but in the JVM, you have to allocate 1-thread for polling these results, which also keeps a connection busy.

What I ended up doing is leveraging akka-stream to stream the queue data directly from the db, which makes it simple to define throttling rules, this is super simple and effective.

Re: Choose Postgres queue technology

#164

Post by johnthescott »

aduffy wrote: post #2

For several projects I’ve opted for the even dumber approach, that works out of the box with every ORM/Query DSL framework in every language: using a normal table with SELECT FOR UPDATE SKIP LOCKED https://www.pgcasts.com/episodes/the-skip-locked-feature-in-... It’s not “web scale” but it easily extends to several thousand background jobs in my experience

skip lock works well on many Ks/sec message queues.

Re: Choose Postgres queue technology

#165

Post by arthurbrown »

dools wrote: post #42

I'm always surprised that when I see people talk about queues I never see anyone mention beanstalkd. I've been using it for basically everything for 10 years and it's solid as a rock, incredibly simple and requires basically no maintenance. It Just Works™

I've used beanstalkd for personal projects previously with a similar experience to you.

For one such project, the message 'priority' feature was a life saver and a feature that is not super common in competing solutions.

Re: Choose Postgres queue technology

#166

Post by j16sdiz »

lmm wrote: post #140
mianos wrote: post #135

Earlier quoted context omitted.

> It works great until it doesn't, and the way it breaks puts you in a state that's very difficult to recover from. This is similar to saying, 'if I mess up all the tables in one database I wreck the rest'. Just my opinion, but this is not actually a thing in databases. Maybe compromised the performance of one database due to another loading things up? I think database are developed with this as an important consider…

> This is similar to saying, 'if I mess up all the tables in one database I wreck the rest'. Just my opinion, but this is not actually a thing in databases. If you mess up the tables in one database it doesn't affect others, but if you lock up the server where it can't respond to queries, that affects every database running on that server. > Also, if you run one postgres, you won't have of an issue running another if…

> If you mess up the tables in one database it doesn't affect others, but if you lock up the server where it can't respond to queries, that affects every database running on that server.

How is it different from: putting multiple queues on same redis, when one queue is locked up, others queue are affected?

If that's a real risk, you can always put them into different instances. The solution is exactly the same for redis or postgresql

Re: Choose Postgres queue technology

#167

Post by yxhuvud »

pqdbr wrote: post #161
phamilton wrote: post #151

Earlier quoted context omitted.

I can share our experience with RabbitMQ/SQS/Sidekiq. Our two major issues have been around the retry mechanism and resource bottlenecks. The key retry problem is "What happens when a worker crashes?". RabbitMQ solves this problem by tying "unacknowledged messages" to a tcp connection. If the connection dies, the in-flight messages are made available to other connections. This is a decent approach, but we hit a lot o…

> Sidekiq solves "What happens when a worker crashes?" by just not solving it. In the free version, those jobs are just lost. I've been using Sidekiq for 11+ years in production and I've never seen this happen. Sidekiq (free version) has a very robust retry workflow. What are you talking about here?

He is talking about the case when the worker itself die for some reason. It can be due to for example when the worker died due to using too much memory or if it hits a segfault or whatever.

Re: Choose Postgres queue technology

#168

Post by maxbond »

valzam wrote: post #85
Ozzie_osman wrote: post #84

One thing I love about Kafka is... It's just an append-only log, and a client is essentially just holding an offset. This is conceptually very simple to reason about. It's also persistent and pretty fault-tolerant (you can just go back and read any offset). Unfortunately, Kafka carries with it enough complexity (due to the distributed nature) that it ends up not being worth it for most use-cases. Personally I'd love…

Considering that you have a native "offset" (auto incrementing id) and the ability to partition by date I would say postgres is a great candidate for a simple Kafka replacement. It will also be significantly simpler to set up consumers if you don't really need to whole consumer group, partition etc. functionality.

Unfortunately `serial` is not sufficient on it's own for that use case. If you observe the values 1, 2, and 4, you can't actually conclude whether 3 exists or not. That transaction may have failed after incrementing the serial, meaning 3 does not exist, or it may be an ongoing transaction which hasn't yet committed, meaning 3 exists but is not yet visible to you.

So if you update your offset to 4 before the transaction for 3 commits, you'll lose 3 forever (unless you control for this, eg by periodically detecting & requeueing orphaned jobs, or by using a strictly serial/gap free integer rather than the built in auto incrementing type).

Re: Choose Postgres queue technology

#169

Post by latchkey »

donatj wrote: post #155
mr-karan wrote: post #153

Earlier quoted context omitted.

> The workers ask for work when they want it, rather than being constantly listening Can you elaborate more on this? How do the workers know when they have to process a new job? Also, am I right in assuming this is typically a single node setup only, as all the files are mounted on a non "share-able" XFS disk?

They ask for work after they finish the previous job (or jobs, they can ask for more than one). Each worker is a single process built just for one task. If there's no work for them there's a small timeout and they ask for more. Simple loop. It's all part of a library we built for building workers. For better or worse, it's all done over http. You are right, though, it is one XFS volume per queue instance. We just run…

I like how GCP cloud tasks reverses the model. Instead of workers pinging the server asking for work, have the queue ping the worker and the worker is effectively a http endpoint. So you send a message to the server, it queues it and then pings a worker with the message.

https://cloud.google.com/tasks/docs/dual-overview

Re: Choose Postgres queue technology

#170

Post by Rapzid »

vhiremath4 wrote: post #78

We used postgres for some of our queues back when we were at ~10 msg/s. It scaled quite a bit, but, honestly, setting up SQS or some other queue stack in AWS, GCP, or Azure is so simple and purpose built for the task (with DL queues and the like built in), I don’t know why you wouldn’t just go that route and not have to worry about that system shitting the bed and affecting the rest of the DB’s health. It seems fooli…

> I don’t know why you wouldn’t just go that route and not have to worry about that system shitting the bed

Because different software has different requirements. Not having an external service requirement other than Postgres might be a feature of an on-prem/b2b appliance.

Because some software may be projected never outgrow the capabilities of Postgres, and if it does moving to another service can be made very easy.

Because you want a transitional job system and the simplicity of doing it in Postgres.

Post reply on HN