Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oban, the job processing framework from Elixir, has come to Python

dimamik.com

Post reply on HN
Print view View original

11–20 of 107 posts

Re: Oban, the job processing framework from Elixir, has come to Python

#11

Post by simonw »

> Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back.

This is such a key feature. Lots of people will tell you that you shouldn't use a relational database as a worker queue, but they inevitably miss out on how important transactions are for this - it's really useful to be able to say "queue this work if the transaction commits, don't queue it if it fails".

Brandur Leach wrote a fantastic piece on this a few years ago: https://brandur.org/job-drain - describing how, even if you have a separate queue system, you should still feed it by logging queue tasks to a temporary database table that can be updated as part of those transactions.

Re: Oban, the job processing framework from Elixir, has come to Python

#12

Post by sanswork »

owaislone wrote: post #9

I don't know how I feel about free open source version and then a commercial version that locks features. Something inside me prevents me from even trying such software. Logically I'd say I support the model because open source needs to be sustainable and we need good quality developer tools and software but when it comes to adoption, I find myself reaching for purely open source projects. I think it has to do with f…

For most of the history the main locked feature was just a premium web interface(there were a few more but that was the main draw) that's included in free now and I think the locked features are primarily around most specialised job ordering engines. Things that if you need free you almost certainly don't need. Oban has been very good about deciding what features to lock away.

(I've paid for it for years despite not needing any of the pro features)

Re: Oban, the job processing framework from Elixir, has come to Python

#13

Post by BowBun »

hangonhn wrote: post #4

This is something my company has been considering for a while. We've been using celery and it's not great. It gets the job done but it has its issue. I've never heard of Oban until now and the one we've considered was Temporal but that feels so much more than what we need. I like how light Oban is. Does anyone have experience with both and is able to give a quick comparison? Thanks!

Very, very different tools, though they cover similar areas.

Temporal - if you have strict workflow requirements, want _guarantees_ that things complete, and are willing to take on extra complexity to achieve that. If you're a bank or something, probably a great choice.

Oban - DB-backed worker queue, which processes tasks off-thread. It does not give you the guarantees that Temporal can because it has not abstracted every push/pull into a first-class citizen. While it offers some similar features with workflows, to multiple 9's of reliability you will be hardening that yourself (based on my experience with Celery+Sidekiq)

Based on my heavy experience with both, I'd be happy to have both available to me in a system I'm working on. At my current job we are forced to use Temporal for all background processing, which for small tasks is just a lot of boilerplate.

Re: Oban, the job processing framework from Elixir, has come to Python

#14

Post by simonw »

semiquaver wrote: post #5
mperham wrote: post #2

I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this! I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central se…

Isn’t it more accurate to say that they are both based on Resque?

Sidekiq credits BackgrounDRb and Delayed::Job and Resque as inspiration here: https://www.mikeperham.com/2022/01/17/happy-10th-birthday-si...

Re: Oban, the job processing framework from Elixir, has come to Python

#15

Post by owaislone »

hangonhn wrote: post #4

This is something my company has been considering for a while. We've been using celery and it's not great. It gets the job done but it has its issue. I've never heard of Oban until now and the one we've considered was Temporal but that feels so much more than what we need. I like how light Oban is. Does anyone have experience with both and is able to give a quick comparison? Thanks!

I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?

Re: Oban, the job processing framework from Elixir, has come to Python

#16

Post by pawelduda »

dfajgljsldkjag wrote: post #8

I have fixed many broken systems that used redis for small tasks. It is much better to put the jobs in the database we already have. This makes the code easier to manage and we have fewer things to worry about. I hope more teams start doing this to save time.

In Rails at least,aside from being used for background processing, redis gives you more goodies. You can store temporary state for tasks that require coordination between multiple nodes without race conditions, cache things to take some load off your DB, etc.

Besides, DB has higher likehood of failing you if you reach certain throughputs

Re: Oban, the job processing framework from Elixir, has come to Python

#17

Post by brightball »

BowBun wrote: post #10
dfajgljsldkjag wrote: post #8

I have fixed many broken systems that used redis for small tasks. It is much better to put the jobs in the database we already have. This makes the code easier to manage and we have fewer things to worry about. I hope more teams start doing this to save time.

Traditional DBs are a poor fit for high-throughput job systems in my experience. The transactions alone around fetching/updating jobs is non-trivial and can dwarf regular data activity in your system. Especially for monoliths which Python and Ruby apps by and large still are. Personally I've migrated 3 apps _from_ DB-backed job queues _to_ Redis/other-backed systems with great success.

The way that Oban for Elixir and GoodJob for Ruby leverage PostgreSQL allows for very high throughput. It's not something that easily ports to other DBs.

Re: Oban, the job processing framework from Elixir, has come to Python

#18

Post by owaislone »

brightball wrote: post #17
BowBun wrote: post #10

Earlier quoted context omitted.

Traditional DBs are a poor fit for high-throughput job systems in my experience. The transactions alone around fetching/updating jobs is non-trivial and can dwarf regular data activity in your system. Especially for monoliths which Python and Ruby apps by and large still are. Personally I've migrated 3 apps _from_ DB-backed job queues _to_ Redis/other-backed systems with great success.

The way that Oban for Elixir and GoodJob for Ruby leverage PostgreSQL allows for very high throughput. It's not something that easily ports to other DBs.

Interesting. Any docs that explain what/how they do this?

Re: Oban, the job processing framework from Elixir, has come to Python

#19

Post by ai_critic »

mperham wrote: post #2

I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this! I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central se…

"based on" is sorta a stretch here.

Sidekiq is pretty bare bones compared to what Oban supports with workflows, crons, partitioning, dependent jobs, failure handling, and so forth.

Re: Oban, the job processing framework from Elixir, has come to Python

#20

Post by TkTech »

The Oban folks have done amazing, well-engineered work for years now - it's really the only option for Elixir. That said, I'm very confused at locking the process pool behind a pro subscription - this is basic functionality given CPython's architecture, not a nice-to-have.

For $135/month on Oban Pro, they advertise:

    All Open Source Features

    Multi-Process Execution

    Workflows

    Global and Rate Limiting

    Unique Jobs

    Bulk Operations

    Encrypted Source (30/90-day refresh)

    1 Application

    Dedicated Support
I'm going to toot my own horn here, because it's what I know, but take my 100% free Chancy for example - https://github.com/tktech/chancy. Out of the box the same workers can mix-and-match asyncio, processes, threads, and sub-interpreters. It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues). There are many more options available in the Python ecosystem than Elixir, so you're competing against Temporal, Trigger, Prefect, Dagster, Airflow, etc etc.
Post reply on HN