Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

github.com

Post reply on HN
Print view View original

51–60 of 140 posts

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#51

Post by memset »

GordonS wrote: post #10
memset wrote: post #7

Earlier quoted context omitted.

I am building an SQS compatible queue for exactly that reason. Use with any language or framework. https://github.com/poundifdef/smoothmq It is based on SQLite, but it’s written in a modular way. It would be easy to add Postgres as a backend (in fact, it might “just work” if I switch the ORM connection string.)

Does SmoothMQ support running multiple nodes for high availability? (I didn't see anything in the docs, but they seem unfinished)

Not today. It's a work in progress! There are several iterations that I'm working on:

1. Primary with secondaries as replicas (replication for availability) 2. Sharding across multiple nodes (sharding for horizontal scaling) 3. Sharding with replication

However, those aren't ready yet. The easiest way to implement this would probably be to use Postgres as the backing storage for the queue, which means relying on Postgres' multiple node support. Then the queue server itself could also scale up and down independently.

Working on the docs! I'd love your feedback - what makes them seem unfinished? (What would you want to see that would make them feel more complete?)

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#52

Post by jmvoodoo »

mind-blight wrote: post #42
earthnail wrote: post #13

Earlier quoted context omitted.

This would be so immensely useful. I’d estimate that there are so many cases where the producer is Node or Rails and the consumer is Python.

This is the exact use case I'm running into right now. I've been looking at BullMQ some it has good typescript support, and is working towards a 1.0 for python. But, I have tried it out in a production stack yet

We have been using bullmq in production for just over a year. It is a piece of technology that our team doesn't have to think about, which is pretty much all I could ask for.

We did end up adding some additional generics which allows us to get strong typing between producers and consumers. That I think has been a key piece of making it easy to use and avoiding dumb mistakes.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#53

Post by whateveracct »

samwillis wrote: post #2

This looks like a great task queue, I'm a massive proponent of "Postgres is all you need" [0] and doubling down on it with my project that takes it to the extreme. What I would love is a Postgres task queue that does multi-step pipelines, with fan out and accumulation. In my view a structured relational database is a particularly good backend for that as it inherently can model the structure. Is that something you ha…

I've come to hate "Postgres is all you need." Or at least, "a single Postgres database is all you need."

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#54

Post by agrothberg »

memset wrote: post #50
justinclift wrote: post #48

Earlier quoted context omitted.

Sounds like it wouldn't have immediate notification of new submissions due to no listen/notify support in SQLite?

It does not implement immediate notification of new submissions because the SQS protocol doesn't have a "push" mechanism, only pull. The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement…

SQS does offer long polling, which looks closer to "push" semantics.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#56

Post by memset »

agrothberg wrote: post #54
memset wrote: post #50

Earlier quoted context omitted.

It does not implement immediate notification of new submissions because the SQS protocol doesn't have a "push" mechanism, only pull. The software, however, could support this for a different queue protocol. This is because SQLite is just used as a disk store for queue items. The golang code itself still processes each message before writing to disk. Since that code is "aware" of incoming messages, it could implement…

SQS does offer long polling, which looks closer to "push" semantics.

Fair enough. I do implement long polling!

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#57

Post by topspin »

nsonha wrote: post #45
rtpg wrote: post #40

I am going to go the other direction on this... to anyone reading this, please consider using a backend-generic queueing system for your Python project. Why? Mainly because those systems offer good affordances for testing and running locally in an operationally simple way. They also tend to have decent default answers for various futzy questions around disconnects at various parts of the workflow. We all know Celery…

> those systems offer good affordances for testing and running locally in an operationally simple way Define "operationally simple", most if not all of them need persistent anyway, on top of the queue itself. This eliminates the queue and uses a persistent you likely already have.

Some message queue brokers that traditionally implement their own backends can also use Postgresql (and other RDBMSs) for persistence. This is a reasonable option if you a.) want to consolidate persistence backends b.) want a mature, battle proven broker and client stack.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#59

Post by AlphaSite »

cklee wrote: post #4

I’ve been thinking about the potential for PostgreSQL-backed job queue libraries to share a common schema. For instance, I’m a big fan of Oban in Elixir: https://github.com/sorentwo/oban Given that there are many Sidekiq-compatible libraries across various languages, it might be beneficial to have a similar approach for PostgreSQL-based job queues. This could allow for job processing in different languages while main…

River is my go to in Golang, it’s really handy to have transactional queuing with a nice little ui.

Re: Show HN: PgQueuer – Transform PostgreSQL into a Job Queue

#60

Post by jashmatthews »

samwillis wrote: post #2

This looks like a great task queue, I'm a massive proponent of "Postgres is all you need" [0] and doubling down on it with my project that takes it to the extreme. What I would love is a Postgres task queue that does multi-step pipelines, with fan out and accumulation. In my view a structured relational database is a particularly good backend for that as it inherently can model the structure. Is that something you ha…

Putting low throughput queues in the same DB is great both for simplicity and for getting exactly-once-processing.

Putting high throughput queues in Postgres sucks because...

No O(1) guarantee to get latest job. Query planner can go haywire.

High update tables bloat like crazy. Needs a whole new storage engine aka ZHEAP

Write amplification as every update has to update every index

LISTEN/NOTIFY doesn't work through connection pooling

Post reply on HN