131–140 of 140 posts
We use listen notify extensively and it is great. The things it lacks most for us is guaranteed single recipient. All subscribers get all notifications which leads to problems in determining who should act on the message n our case.
Earlier quoted context omitted.
Thats exactly what we do but taking a lock takes 1 RTT to the database which means about 100ms. it limits the number of events receivers can handle. IF you have too many events, receivers will be just trying to take a lock most of the time.
Of my head, you could attach a uuid or sequence number to the emitted event. Then based on the uuid or sequence you can let one or the other event consumer pick? Ex. you have two consumers, if the sequence number is odd, A picks it, if its even B picks.
Earlier quoted context omitted.
Thats exactly what we do but taking a lock takes 1 RTT to the database which means about 100ms. it limits the number of events receivers can handle. IF you have too many events, receivers will be just trying to take a lock most of the time.
Of my head, you could attach a uuid or sequence number to the emitted event. Then based on the uuid or sequence you can let one or the other event consumer pick? Ex. you have two consumers, if the sequence number is odd, A picks it, if its even B picks.
Earlier quoted context omitted.
Of my head, you could attach a uuid or sequence number to the emitted event. Then based on the uuid or sequence you can let one or the other event consumer pick? Ex. you have two consumers, if the sequence number is odd, A picks it, if its even B picks.
Without more logic than this does this mean any workers that are busy or down mean you lose jobs?
We use listen notify extensively and it is great. The things it lacks most for us is guaranteed single recipient. All subscribers get all notifications which leads to problems in determining who should act on the message n our case.
Wouldn't it be possible for you to lose jobs if no subscriber is listening?
Earlier quoted context omitted.
Thats exactly what we do but taking a lock takes 1 RTT to the database which means about 100ms. it limits the number of events receivers can handle. IF you have too many events, receivers will be just trying to take a lock most of the time.
Of my head, you could attach a uuid or sequence number to the emitted event. Then based on the uuid or sequence you can let one or the other event consumer pick? Ex. you have two consumers, if the sequence number is odd, A picks it, if its even B picks.
Earlier quoted context omitted.
Some names - Celery (massive and heavy) - Dramatiq - APScheduler - Huey Today, Redis queues, unless stricly a single process, seem to be most pain free for small scale use.
We had a terrible time with Dramatiq; very buggy and resource-heavy. We ended up switching to SNS/SQS combo
I think most people reading this site are working on relatively small systems (that still need background tasks!) and the fixed costs of background tasks can be reasonable. But I could be totally offbase
I really like the emergence of simple queuing tools for robust database management systems. Keep things simple and remove infrastructure complexity. Definitely a +1 from me! For handling straightforward asynchronous tasks like sending opt-in emails, we've developed a similar library at All Quiet for C# and MongoDB: https://allquiet.app/open-source/mongo-queueing In this context: LISTEN/NOTIFY in PostgreSQL is compara…