https://tkte.ch/chancy/https://github.com/tktech/chancy
Chancy is a postgres-backed task queue for Python that scratches a lot of the Celery itches. It's not meant for folks that need to run a million tasks a second, but for the majority of projects (many millions per day) it offers:
- Robust job support, including timeouts, memory limits, retries, global uniqueness, global rate limiting, scheduling (cron and "in 10 seconds"), priorities, etc...
- DAG based workflow plugin
- It's asyncio-first with support for threading, multiprocessing, asyncio, and sub-interpreter tasks so each job can use the optimal concurrency model. Workflows can mix tasks across multiple queues and concurrency models.
- Can be embedded inside your existing ASGI servers - great for things like development docker containers or containers deployed on say, unraid.
- Worker's handle scheduling (no need for `celery beat`) and have an optional built-in dashboard.
- 1 infrastructure dependency (postgres) and 1 required package (psycopg3) - with everything else isolated in optional plugins.
- Dynamically re-assign queues to new workers based on tags, add, pause (with auto-resume), modify or destroy queues at any time.
- Highly observable - unlike Celery, you can just query your database when needed to see the entire system state
- Portable - Linux/Windows/OSX
- Permanently free and open-source without any "premium" or paid features.
- Django integration - ORM/models, admin, and django auth integration for the dashboard.
Chancy is a young tool, but is used in production environments with tens of thousands of users and billions of jobs run with great feedback from early users:
> ...thank you for this amazing library...
> hey, first thanks for such a great library! Chancy has worked incredibly well, and its modular design has made it a pleasure to use. Super lightweight but feature-full; it's a hard balance to strike.