Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Choose Postgres queue technology

adriano.fyi

Post reply on HN
Print view View original

171–180 of 369 posts

Re: Choose Postgres queue technology

#171

Post by imiric »

phamilton wrote: post #151
reubenbond wrote: post #113

Earlier quoted context omitted.

> Anything but RabbitMQ. Would you mind elaborating on this? I'd be happy for others to chime in with their experiences/opinions, too.

I can share our experience with RabbitMQ/SQS/Sidekiq. Our two major issues have been around the retry mechanism and resource bottlenecks. The key retry problem is "What happens when a worker crashes?". RabbitMQ solves this problem by tying "unacknowledged messages" to a tcp connection. If the connection dies, the in-flight messages are made available to other connections. This is a decent approach, but we hit a lot o…

That's very insightful, thanks for sharing.

Do you have any experience with NATS, and how would you compare it to RMQ/SQS?

The authors claim it guarantees exactly-once delivery with its JetStream component, and it looks very alluring from the documentation, but looks can be deceiving.

Re: Choose Postgres queue technology

#173

Post by donatj »

latchkey wrote: post #169
donatj wrote: post #155

Earlier quoted context omitted.

They ask for work after they finish the previous job (or jobs, they can ask for more than one). Each worker is a single process built just for one task. If there's no work for them there's a small timeout and they ask for more. Simple loop. It's all part of a library we built for building workers. For better or worse, it's all done over http. You are right, though, it is one XFS volume per queue instance. We just run…

I like how GCP cloud tasks reverses the model. Instead of workers pinging the server asking for work, have the queue ping the worker and the worker is effectively a http endpoint. So you send a message to the server, it queues it and then pings a worker with the message. https://cloud.google.com/tasks/docs/dual-overview

Ooh, that's kind of interesting. Am I reading this right that it holds the HTTP connection open for up to thirty minutes waiting for the work to complete? That's kind of wild.

Re: Choose Postgres queue technology

#174

Post by physicles »

what-no-tests wrote: post #101
beachy wrote: post #92

Earlier quoted context omitted.

Our industry is full of shysters pushing their own technology. Time and again, it turns out that a RDBMS will handle that job just fine. That's really the premise of this article. So please, go on and back up your bold statements with some specifics. Why specifically is it not OK to use a database as a message queue?

No time at the moment to break it all down, but here's a previous discussion thread on a similar topic. https://news.ycombinator.com/item?id=27483003

Good counter-point. I see these main points over there:

1. It doesn't scale (there it is again)

2. Queuing with Postgres is super fiddly to get right

3. You're hacking a queue on top of something that isn't a queue

4. Running redis or rabbit isn't all that complicated

#1 as TFA argues, premature concern about scaling is the root of so much needless complexity. You should make scaling decisions like this: 1) assume boring tech like PG will satisfy your needs; 2) if it demonstrably does not, then find something that does.

#2 is obviously true; just look at this thread. There are battle-tested queuing libraries in most popular languages, but you do have to dig into the details of how they interact with things like pgbouncer.

#3 I guess so? But if the queue abstraction works and isn't leaky, what does it matter?

#4 can be debated. For several years I've been running a moderately complicated setup with 2 databases, redis, and kafka for several years. There's no way I'm going to add another piece of tech unless there's no other choice. The cognitive cost is too high.

The main debate is in the tradeoff between #2 and #4. Personally, if I can use an existing piece of tech to solve a problem to avoid having to ops another piece of tech, then I'm going to do that every time.

Re: Choose Postgres queue technology

#175

Post by latchkey »

donatj wrote: post #173
latchkey wrote: post #169

Earlier quoted context omitted.

I like how GCP cloud tasks reverses the model. Instead of workers pinging the server asking for work, have the queue ping the worker and the worker is effectively a http endpoint. So you send a message to the server, it queues it and then pings a worker with the message. https://cloud.google.com/tasks/docs/dual-overview

Ooh, that's kind of interesting. Am I reading this right that it holds the HTTP connection open for up to thirty minutes waiting for the work to complete? That's kind of wild.

Indeed. If you're hitting AppEngine or GCP Functions, they auto scale workers up for you to manage long running tasks. Ideally though, you finish as quickly as possible by breaking the work down into more tasks. That way, you can parallelize as much as possible.

It is all configurable, but I've scaled up to hundreds of workers at a time to blast through tasks and it wasn't expensive at all.

Workers being an HTTP endpoint makes them super easy to implement and even better... write tests for.

Re: Choose Postgres queue technology

#176

Post by kureikain »

imiric wrote: post #171
phamilton wrote: post #151

Earlier quoted context omitted.

I can share our experience with RabbitMQ/SQS/Sidekiq. Our two major issues have been around the retry mechanism and resource bottlenecks. The key retry problem is "What happens when a worker crashes?". RabbitMQ solves this problem by tying "unacknowledged messages" to a tcp connection. If the connection dies, the in-flight messages are made available to other connections. This is a decent approach, but we hit a lot o…

That's very insightful, thanks for sharing. Do you have any experience with NATS, and how would you compare it to RMQ/SQS? The authors claim it guarantees exactly-once delivery with its JetStream component, and it looks very alluring from the documentation, but looks can be deceiving.

It has multiple mode. One of them is explicitly acknowlede mode. If the worker finished process the job but doesn't ack, the message will appear again.

Re: Choose Postgres queue technology

#177

Post by boyter »

reubenbond wrote: post #113
mianos wrote: post #91

Skype used postgres as queue with a small plugin to process all their CDR many years ago. I have no idea if it used these days but it was 'web scale', 10 years ago. Just working, while people on the internet argued about using a database as a queue is an anti-pattern. Having transactions is quite handy. https://wiki.postgresql.org/wiki/SkyTools I did a few talks on this at Sydpy as I used it at work quite a bit. It's…

> Anything but RabbitMQ. Would you mind elaborating on this? I'd be happy for others to chime in with their experiences/opinions, too.

I used RabbitMQ for a while and nothing but problems.

Admittedly I probably shouldn't have used it the way I did. I dumped many millions of tasks into it, then fanned out processes pulling from that queue that took a variable amount of time to run. Some ran in seconds, some hours.

I had picked RabbitMQ because I wanted that queue to be durable and resist workers dying, or being restarted. However long lived tasks like this is not really what it was designed for (in my opinion). I kept running into issues where it would take a long time to restart, and stop answering connections and need a restart to continue. I ended up having to write monitoring code to check for this and handle it to have it be slightly reliable.

Im sure it works well for smaller short lived messages, but considering the issues I bumped into I would be hesitant to try it. Id probably reach to redis first with wrappers allowing me to swap out to any other queue as required first.

Re: Choose Postgres queue technology

#178

Post by maxbond »

surprisetalk wrote: post #112
thom wrote: post #110

Earlier quoted context omitted.

Presumably it's okay that this loses work if your task runner has an error?

If you read my guide, you’ll see that I embed it in a transaction that doesn’t COMMIT until the companion code is complete :) For example, I run the above query to grab a queued email, send it using mailgun, then COMMIT. Nothing is changed in the DB unless the email is sent.

Holding a transaction open for the duration of a request to an external service makes me nervous. I've seen similar code lock up the database and bring down production. Are you using timeouts and circuit breakers to control the length of the transactions?

Re: Choose Postgres queue technology

#179

Post by djur »

paulddraper wrote: post #73
klysm wrote: post #70

Earlier quoted context omitted.

Not all use cases are high throughput. That’s not what makes it a toy

"Toy/low throughput" = "Toy or low throughout"

Can you define "low throughput"? I think people have significantly different ideas of what that means.

Re: Choose Postgres queue technology

#180

Post by ansc »

zacksiri wrote: post #148

During my tenure as CTO at a fintech company I built a banking engine using postgres backed queue system using Elixir / Phoenix. It's still in use today. The company processed large volumes of transactions and we were able to do things in real-time in terms of payments. Our system reached a point where I realized that we can scale almost infinitely just using a 2 tier architecture (Elixir / Phoenix / Oban and Postgre…

>There was a lot of back and forth between engineers that discussed whether we should add the index.

Jeez. What was the idea behind not adding? Disk space I presume?

Post reply on HN