Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oban, the job processing framework from Elixir, has come to Python

dimamik.com

Post reply on HN
Print view View original

21–30 of 107 posts

Re: Oban, the job processing framework from Elixir, has come to Python

#21

Post by TkTech »

owaislone wrote: post #15
hangonhn wrote: post #4

This is something my company has been considering for a while. We've been using celery and it's not great. It gets the job done but it has its issue. I've never heard of Oban until now and the one we've considered was Temporal but that feels so much more than what we need. I like how light Oban is. Does anyone have experience with both and is able to give a quick comparison? Thanks!

I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?

Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.

Re: Oban, the job processing framework from Elixir, has come to Python

#23

Post by brightball »

owaislone wrote: post #18
brightball wrote: post #17

Earlier quoted context omitted.

The way that Oban for Elixir and GoodJob for Ruby leverage PostgreSQL allows for very high throughput. It's not something that easily ports to other DBs.

Interesting. Any docs that explain what/how they do this?

Good Job is a strong attempt. I believe it's based around Advisory Locks though.

https://github.com/bensheldon/good_job

Re: Oban, the job processing framework from Elixir, has come to Python

#24

Post by TkTech »

owaislone wrote: post #18
brightball wrote: post #17

Earlier quoted context omitted.

The way that Oban for Elixir and GoodJob for Ruby leverage PostgreSQL allows for very high throughput. It's not something that easily ports to other DBs.

Interesting. Any docs that explain what/how they do this?

A combination of LISTEN/NOTIFY for instantaneous reactivity, letting you get away with just periodic polling, and FOR UPDATE...SKIP LOCKED making it efficient and safe for parallel workers to grab tasks without co-ordination. It's actually covered in the article near the bottom there.

Re: Oban, the job processing framework from Elixir, has come to Python

#25

Post by hangonhn »

TkTech wrote: post #21
owaislone wrote: post #15

Earlier quoted context omitted.

I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?

Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.

Yeah that list right there. That's exactly it.

We don't hate Celery at all. It's just a bit harder to get it to do certain things and requires a bit more coding and understanding of celery than what we want to invest time and effort in.

Again, no hate towards Celery. It's not bad. We just want to see if there are better options out there.

Re: Oban, the job processing framework from Elixir, has come to Python

#26

Post by sorentwo »

TkTech wrote: post #20

The Oban folks have done amazing, well-engineered work for years now - it's really the only option for Elixir. That said, I'm very confused at locking the process pool behind a pro subscription - this is basic functionality given CPython's architecture, not a nice-to-have. For $135/month on Oban Pro, they advertise: All Open Source Features Multi-Process Execution Workflows Global and Rate Limiting Unique Jobs Bulk O…

> It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues).

We may well move some of those things to the OSS version, depending on interest, usage, etc. It's much easier to make things free than the other way around. Some Pro only features in Elixir have moved to OSS previously, and as a result of this project some additional functionality will also be moved.

Support only options aren't going to cut it in our experience; but maybe that'll be different with Python.

> There are many more options available in the Python ecosystem than Elixir, so you're competing against Temporal, Trigger, Prefect, Dagster, Airflow, etc etc.

There's a lot more of everything available in the Python ecosystem =)

Re: Oban, the job processing framework from Elixir, has come to Python

#27

Post by owaislone »

TkTech wrote: post #24
owaislone wrote: post #18

Earlier quoted context omitted.

Interesting. Any docs that explain what/how they do this?

A combination of LISTEN/NOTIFY for instantaneous reactivity, letting you get away with just periodic polling, and FOR UPDATE...SKIP LOCKED making it efficient and safe for parallel workers to grab tasks without co-ordination. It's actually covered in the article near the bottom there.

Thank you

Re: Oban, the job processing framework from Elixir, has come to Python

#28

Post by alanwreath »

owaislone wrote: post #15
hangonhn wrote: post #4

This is something my company has been considering for a while. We've been using celery and it's not great. It gets the job done but it has its issue. I've never heard of Oban until now and the one we've considered was Temporal but that feels so much more than what we need. I like how light Oban is. Does anyone have experience with both and is able to give a quick comparison? Thanks!

I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?

I like celery but I started to try other things when I had projects doing work from languages in addition to python. Also I prefer the code work without having to think about queues as much as possible. In my case that was Argo workflows (not to be confused with Argo CD)

Re: Oban, the job processing framework from Elixir, has come to Python

#29

Post by qianli_cs »

Thanks for sharing, interesting project! One thing that stood out to me is that some fairly core features are gated behind a Pro tier. For context, there are prior projects in this space that implement similar ideas fully in OSS, especially around Postgres-backed durable execution:

1. DBOS built durable workflows and queues on top of Postgres (disclaimer: I'm a co-founder of DBOS), with some recent discussions here: https://news.ycombinator.com/item?id=44840693

2. Absurd explores a related design as well: https://news.ycombinator.com/item?id=45797228

Overall, it's encouraging to see more people converging on a database-centric approach to durable workflows instead of external orchestrators. There's still a lot of open design space around determinism, recovery semantics, and DX. I'm happy to learn from others experimenting here.

Re: Oban, the job processing framework from Elixir, has come to Python

#30

Post by sorenone »

mperham wrote: post #2

I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this! I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central se…

Thanks Mike! You are an inspiration. Parker and I have different strengths both in life and language. We're committed to what this interop brings to both Python and Elixir.
Post reply on HN