121–130 of 140 posts
I really like the emergence of simple queuing tools for robust database management systems. Keep things simple and remove infrastructure complexity. Definitely a +1 from me! For handling straightforward asynchronous tasks like sending opt-in emails, we've developed a similar library at All Quiet for C# and MongoDB: https://allquiet.app/open-source/mongo-queueing In this context: LISTEN/NOTIFY in PostgreSQL is compara…
I've been referring to this post about issues with Celery: https://docs.hatchet.run/blog/problems-with-celery Does PgQueuer address any of them?
At first glans i i see two thins that PgQueuer can. It has native async support, it you can also have sync functions, they will be offloaded to threads via anyio ( https://github.com/janbjorge/PgQueuer/blob/99c82c2d661b2ddfc... ) It has a global rate limit, synced via pg notify.
I am going to go the other direction on this... to anyone reading this, please consider using a backend-generic queueing system for your Python project. Why? Mainly because those systems offer good affordances for testing and running locally in an operationally simple way. They also tend to have decent default answers for various futzy questions around disconnects at various parts of the workflow. We all know Celery…
Some names - Celery (massive and heavy) - Dramatiq - APScheduler - Huey Today, Redis queues, unless stricly a single process, seem to be most pain free for small scale use.
How does LISTEN/NOTIFY compare to using select for update skip locked? I thought listen/notify can lose queue items when the process crashes? Is that true? Do you need to code for those cases in some manner?
Earlier quoted context omitted.
At first glans i i see two thins that PgQueuer can. It has native async support, it you can also have sync functions, they will be offloaded to threads via anyio ( https://github.com/janbjorge/PgQueuer/blob/99c82c2d661b2ddfc... ) It has a global rate limit, synced via pg notify.
PgQueuer seems pretty similar to Hatchet - both use Postgres, require their own worker processes, both support async. Hatchet seems to be a lot more powerful though: cron, DAG workflows, retries, timeouts, and a UI dashboard.
We use listen notify extensively and it is great. The things it lacks most for us is guaranteed single recipient. All subscribers get all notifications which leads to problems in determining who should act on the message n our case.
Any suggestions for something like this for dotnet?
Hangfire with PostgreSQL driver.
We use listen notify extensively and it is great. The things it lacks most for us is guaranteed single recipient. All subscribers get all notifications which leads to problems in determining who should act on the message n our case.
Could use the notify to awake, and then the worker needs to lock the job row? Whichever worker gets the lock, acts on the message
Earlier quoted context omitted.
Could use the notify to awake, and then the worker needs to lock the job row? Whichever worker gets the lock, acts on the message
Thats exactly what we do but taking a lock takes 1 RTT to the database which means about 100ms. it limits the number of events receivers can handle. IF you have too many events, receivers will be just trying to take a lock most of the time.
Ex. you have two consumers, if the sequence number is odd, A picks it, if its even B picks.
Earlier quoted context omitted.
Some names - Celery (massive and heavy) - Dramatiq - APScheduler - Huey Today, Redis queues, unless stricly a single process, seem to be most pain free for small scale use.
We had a terrible time with Dramatiq; very buggy and resource-heavy. We ended up switching to SNS/SQS combo