Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PgQue: Zero-Bloat Postgres Queue

github.com

Post reply on HN
Print view View original

11–20 of 41 posts

Re: PgQue: Zero-Bloat Postgres Queue

#11

Post by mind-blight »

The vacuum pressure is real. Using a system with the skip locked technique + polling caused massive DB perf issues as the queue depth grew. The query to see the current jobs in the queue ended up being the main performance bottleneck, which cause slower throughput, which caused a larger queue depth, which etc.

Scaling the workers sometimes exacerbates the problem because you run into connection limits or polling hammering the DB.

I love the idea of pg as a queue, but I'm a more skeptical of it after dealing with it in production

Re: PgQue: Zero-Bloat Postgres Queue

#13

Post by killingtime74 »

I got Claude to analyze the code and it's not really comparable to SKIP LOCKED queues. It's more like Kafka. There's no job queue semantics with acks, workers taking from same job pool.

It's Kafka like one event stream and multiple independent worker cursors.

It's more SNS than SQS or Kafka than Rabbitmq/Nats

Re: PgQue: Zero-Bloat Postgres Queue

#15

Post by samokhvalov »

killingtime74 wrote: post #13

I got Claude to analyze the code and it's not really comparable to SKIP LOCKED queues. It's more like Kafka. There's no job queue semantics with acks, workers taking from same job pool. It's Kafka like one event stream and multiple independent worker cursors. It's more SNS than SQS or Kafka than Rabbitmq/Nats

correct

it's explained in README:

> Category: River, Que, and pg-boss (and Oban, graphile-worker, solid_queue, good_job) are job queue frameworks. PgQue is an event/message queue optimized for high-throughput streaming with fan-out.

Re: PgQue: Zero-Bloat Postgres Queue

#16

Post by what »

pierrekin wrote: post #4
cout wrote: post #2

I think it's great that projects like this exist where people are building middleware in different ways than others. Still, as someone who routinely uses shared memory queues, the idea of considering a queue built inside a database to be "zero bloat" leaves me scratching my head a bit. I can see why someone would want that, but once person's feature is bloat to someone else.

In Postgres land bloat refers to dead tuples that are left in place during certain operations and need to be vacuumed later. It’s challenging to write a queue that doesn’t create bloat, hence why this project is citing it as a feature.

Can’t you just partition the table by time (or whatever) and drop old partitions and not worry about vacuuming? Why do you need to keep around completed jobs forever?

Re: PgQue: Zero-Bloat Postgres Queue

#17

Post by klysm »

mind-blight wrote: post #11

The vacuum pressure is real. Using a system with the skip locked technique + polling caused massive DB perf issues as the queue depth grew. The query to see the current jobs in the queue ended up being the main performance bottleneck, which cause slower throughput, which caused a larger queue depth, which etc. Scaling the workers sometimes exacerbates the problem because you run into connection limits or polling hamm…

What kind of throughput are we talking about?

Re: PgQue: Zero-Bloat Postgres Queue

#18

Post by BodyCulture »

mind-blight wrote: post #11

The vacuum pressure is real. Using a system with the skip locked technique + polling caused massive DB perf issues as the queue depth grew. The query to see the current jobs in the queue ended up being the main performance bottleneck, which cause slower throughput, which caused a larger queue depth, which etc. Scaling the workers sometimes exacerbates the problem because you run into connection limits or polling hamm…

Is your comment referring to this project specifically?

Because the docs say:

  PgQue avoids that whole class of problems. It uses snapshot-based batching and TRUNCATE-based table rotation instead of per-row deletion.
Would be great if you could specify if you had problems with the exact implementation linked by op or if you did write about a different thing, thanks!

Re: PgQue: Zero-Bloat Postgres Queue

#19

Post by adhocmobility »

Why insist on calling this a queue when it doesn't really have queue semantics? Queues do the job of load balancing between different workers. When workers acknowledge tasks, they get deleted, and there are visibility timeouts.

This is a log.

It's not really solving the problems you claim it solves. It's not, for instance, a replacement for SKIP LOCKED based queues.

Re: PgQue: Zero-Bloat Postgres Queue

#20

Post by pierrekin »

what wrote: post #16
pierrekin wrote: post #4

Earlier quoted context omitted.

In Postgres land bloat refers to dead tuples that are left in place during certain operations and need to be vacuumed later. It’s challenging to write a queue that doesn’t create bloat, hence why this project is citing it as a feature.

Can’t you just partition the table by time (or whatever) and drop old partitions and not worry about vacuuming? Why do you need to keep around completed jobs forever?

Yes you can, and at the risk of sounding a little snarky; if you do something like that and then release it as open source, people may even discuss it on HN!
Post reply on HN