Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

github.com

Post reply on HN
Print view View original

91–100 of 140 posts

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#91

Post by lukebuehler »

_medihack_ wrote: post #17

There is also Procrastinate: https://procrastinate.readthedocs.io/en/stable/index.html Procrastinate also uses PostgreSQL's LISTEN/NOTIFY (but can optionally be turned off and use polling). It also supports many features (and more are planned), like sync and async jobs (it uses asyncio under the hood), periodic tasks, retries, task locks, priorities, job cancellation/aborting, Django integration (optional). DISCLAIME…

I’m using Procrastinate in several projects. Would definitely like to see a comparison.

What I personally love about Procrastinate is async, locks, delayed and scheduled jobs, queue specific workers (allowing to factor the backend in various ways). All this with a simple codebase and schema.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#92

Post by kraihx »

cklee wrote: post #4

I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…

For JavaScript and Perl there is already Minion, which relies on listen/notify + FOR UPDATE SKIP LOCKED.

https://github.com/mojolicious/minion.js

https://github.com/mojolicious/minion

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#94

Post by paffdragon »

jashmatthews wrote: post #60
samwillis wrote: post #2

This looks like a great task queue, I'm a massive proponent of "Postgres is all you need" [0] and doubling down on it with my project that takes it to the extreme. What I would love is a Postgres task queue that does multi-step pipelines, with fan out and accumulation. In my view a structured relational database is a particularly good backend for that as it inherently can model the structure. Is that something you ha…

Putting low throughput queues in the same DB is great both for simplicity and for getting exactly-once-processing. Putting high throughput queues in Postgres sucks because... No O(1) guarantee to get latest job. Query planner can go haywire. High update tables bloat like crazy. Needs a whole new storage engine aka ZHEAP Write amplification as every update has to update every index LISTEN/NOTIFY doesn't work through c…

Indeed, that's my experience too. We used partitions like others mentioned below, but Postgres had issues with moving rows across tables atomically and had to implement our custom complex queries to overcome it. Plus job expiration was dynamic and had to use background cleaning. The bigger problem was with the planner not able to pick up sudden changes in volume and had to use a cron to run analyze on it. Managing retries with backoffs, etc.. At some point we stopped fighting it and just moved to SQS, we have zero problems since, no maintenence needed, and it's still free so we saved storage cost, time and developer effort for ongoing maintenance.

We still use Postgres for simple queues, but those don't really require a library as it's quite simple usually, with some advisory locks we can handle the crashed job unlocking fairly well too.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#96

Post by tmountain »

rileymichael wrote: post #19
cklee wrote: post #4

I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…

If you want a generic queue that can be consumed in any runtime, you can just build it directly into postgres via extensions like https://github.com/tembo-io/pgmq .

Also, pgmq can run as a TLE (trusted language extension), so you can install it into cloud hosted Postgres solutions like Supabase. We're using pgmq, and it's solid so far.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#97

Post by berkes »

whateveracct wrote: post #53
samwillis wrote: post #2

This looks like a great task queue, I'm a massive proponent of "Postgres is all you need" [0] and doubling down on it with my project that takes it to the extreme. What I would love is a Postgres task queue that does multi-step pipelines, with fan out and accumulation. In my view a structured relational database is a particularly good backend for that as it inherently can model the structure. Is that something you ha…

I've come to hate "Postgres is all you need." Or at least, "a single Postgres database is all you need."

I won´t call it "hate", but I've ran into quite some situations where the Postgres version caused a lot of pain.

- When it wasn't as easy as a dedicated solution: where installing and managing a focused service is overall easier than shoehorning it into PG.

- when it didn't perform anywhere close to a dedicated solution: overhead from the guarantees that PG makes (acid and all that) when you don't need them. Or where the relational architecture isn't suited for this type of data: e.g. hierarchical, time-series, etc.

- when it's not as feature complete as a dedicated service: for example I am quite sure one can build (parts of) an ActiveDirectory or Kafka Bus, entirely in PG. But it will lack features that in future you'll likely need - they are built into these dedicated solutions because they are often needed after all.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#98

Post by topspin »

ijustlovemath wrote: post #29

You could even layer in PostgREST for a nice HTTP API that is available from any language!

Already done. See: PostgREST. Want to use PostgreSQL (or most other RDBMSs) as the backend for an actively developed, multiprotocol, multiplatform, open source, battle proven message broker that also provides a messaging REST API of its own? Use ActiveMQ (either flavor) and configure a JDBC backend. Done.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#99

Post by tmountain »

philippta wrote: post #67
whateveracct wrote: post #53

Earlier quoted context omitted.

I've come to hate "Postgres is all you need." Or at least, "a single Postgres database is all you need."

Why?

Just to share an anecdote, we've been able to get to market faster than ever before by just using Postgres (Supabase) for basically everything. We're leveraging RLS, and it's saved us from building an API (just using PostgREST). We've knocked months off our project timeline by doing this.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#100

Post by jeeybee »

odie5533 wrote: post #77

I've been referring to this post about issues with Celery: https://docs.hatchet.run/blog/problems-with-celery Does PgQueuer address any of them?

At first glans i i see two thins that PgQueuer can. It has native async support, it you can also have sync functions, they will be offloaded to threads via anyio (https://github.com/janbjorge/PgQueuer/blob/99c82c2d661b2ddfc...)

It has a global rate limit, synced via pg notify.

Post reply on HN