Looks like a nice API. We have used the similar pattern for years, but with sqlalchemy and the same kind of sql statement for getting the next available job. Think it’s easier to handle worker queues just with postgresql rather than some other queue system to keep supported and updated for security fixes etc.
Oban, the job processing framework from Elixir, has come to Python
61–70 of 107 posts
Re: Oban, the job processing framework from Elixir, has come to Python
#62> Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back. This is such a key feature. Lots of people will tell you that you shouldn't use a relational database as a worker queue, but they inevitably miss out on how important transactions are for th…
I agree this is an awesome feature. I use pg_timetable instead of Oban though: https://cybertec-postgresql.github.io/pg_timetable/v6.x/
Re: Oban, the job processing framework from Elixir, has come to Python
#63I have fixed many broken systems that used redis for small tasks. It is much better to put the jobs in the database we already have. This makes the code easier to manage and we have fewer things to worry about. I hope more teams start doing this to save time.
Traditional DBs are a poor fit for high-throughput job systems in my experience. The transactions alone around fetching/updating jobs is non-trivial and can dwarf regular data activity in your system. Especially for monoliths which Python and Ruby apps by and large still are. Personally I've migrated 3 apps _from_ DB-backed job queues _to_ Redis/other-backed systems with great success.
How high of throughput were you working with? I've used Oban at a few places that had what pretty decent throughput and it was OK. Not disagreeing with your approach at all, just trying to get an idea of what kinds of workloads you were running to compare.
Re: Oban, the job processing framework from Elixir, has come to Python
#64No offense to all of the effort referenced here, I understand that there are many computing contexts with different needs. However, I really need to ask: am I the only one who cringes at the notion of a transactional database being a job processing nexus? Deadlocks anyone? Really sounds like asking for serious trouble to me.
Re: Oban, the job processing framework from Elixir, has come to Python
#65Earlier quoted context omitted.
Traditional DBs are a poor fit for high-throughput job systems in my experience. The transactions alone around fetching/updating jobs is non-trivial and can dwarf regular data activity in your system. Especially for monoliths which Python and Ruby apps by and large still are. Personally I've migrated 3 apps _from_ DB-backed job queues _to_ Redis/other-backed systems with great success.
How high of throughput were you working with? I've used Oban at a few places that had what pretty decent throughput and it was OK. Not disagreeing with your approach at all, just trying to get an idea of what kinds of workloads you were running to compare.
Millions of jobs a minute
Re: Oban, the job processing framework from Elixir, has come to Python
#66Earlier quoted context omitted.
Traditional DBs are a poor fit for high-throughput job systems in my experience. The transactions alone around fetching/updating jobs is non-trivial and can dwarf regular data activity in your system. Especially for monoliths which Python and Ruby apps by and large still are. Personally I've migrated 3 apps _from_ DB-backed job queues _to_ Redis/other-backed systems with great success.
The way that Oban for Elixir and GoodJob for Ruby leverage PostgreSQL allows for very high throughput. It's not something that easily ports to other DBs.
Appreciate the added context here, this is indeed some special sauce that challenges my prior assumptions!
Re: Oban, the job processing framework from Elixir, has come to Python
#67Earlier quoted context omitted.
Traditional DBs are a poor fit for high-throughput job systems in my experience. The transactions alone around fetching/updating jobs is non-trivial and can dwarf regular data activity in your system. Especially for monoliths which Python and Ruby apps by and large still are. Personally I've migrated 3 apps _from_ DB-backed job queues _to_ Redis/other-backed systems with great success.
Transactions around fetching/updating aren't trivial, that's true. However, the work that you're doing _is_ regular activity because it's part of your application logic. That's data about the state of your overall system and it is extremely helpful for it to stay with the app (not to mention how nice it makes testing). Regarding overall throughput, we've written about running one million jobs a minute [1] on a single…
Appreciate the response, I'm learning some new things about the modern listening mechanisms for DBs which unlock more than I believed was possible.
For your first point - I would counter that a lot of data about my systems lives outside of the primary database. There is however an argument for adding a dependency, and for testing complexities. These are by and large solved problems at the scale I work with (not huge, not tiny).
I think both approaches work and I honestly just appreciate you guys holding Celery to task ;)
Re: Oban, the job processing framework from Elixir, has come to Python
#68While this is a Cool Thing To See, I do wish things would go the other way—and have all the BI/ML/DS pipelines and workflows folks are building in Python and have them come to Elixir (and, as would follow, Elixir). I get where the momentum is, but having something functional, fault-tolerant, and concurrent underpinning work that’s naturally highly concurrent and error-prone feels like a _much_ more natural fit.
Agree, and Claude Code does very well with Elixir despite TS/Python getting all the hype:
https://youtu.be/iV1EcfZSdCM?si=KAJW26GVaBqZjR3M
This helps with keeping it on track writing idiomatic elixir and using good patterns: https://skills.sh/agoodway/.claude/elixir-genius
Re: Oban, the job processing framework from Elixir, has come to Python
#69We had considered Oban when deciding whether to go with Kafka/Debezium or not. We sided with Kafka because it can do high throughput ingestion and it is easier to maintain it with cursor in today's world. Postgres is not meant for heavy writes, but heavy querying. You could fix that with lot of care but then it does not scale multi-master very well either. Kafka scales much better for heavy writes.
Re: Oban, the job processing framework from Elixir, has come to Python
#70Is Postgres fast enough for job processing these days? We do hundreds of millions of jobs now and even years ago when our volume was a fraction of that, we got a huge performance boost moving from Postgres + Que to Redis + Sidekiq. Has that changed in the intervening years?