Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oban, the job processing framework from Elixir, has come to Python

dimamik.com

Post reply on HN
Print view View original

101–107 of 107 posts

Re: Oban, the job processing framework from Elixir, has come to Python

#101

Post by TkTech »

xmcqdpt2 wrote: post #95
TkTech wrote: post #21

Earlier quoted context omitted.

Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.

For an even lighter system than Celery, I'm a big fan of https://python-rq.org/ It's super low on the dependencies and integrates nicely as a library into python applications. It's very bare bones.

I kinda have my own https://github.com/tktech/chancy :P

Re: Oban, the job processing framework from Elixir, has come to Python

#102

Post by owaislone »

TkTech wrote: post #20

The Oban folks have done amazing, well-engineered work for years now - it's really the only option for Elixir. That said, I'm very confused at locking the process pool behind a pro subscription - this is basic functionality given CPython's architecture, not a nice-to-have. For $135/month on Oban Pro, they advertise: All Open Source Features Multi-Process Execution Workflows Global and Rate Limiting Unique Jobs Bulk O…

Thanks for sharing Chancy. Looks really interesting. Outside of the API breaking, would you say this is safe enough to for production today?

Re: Oban, the job processing framework from Elixir, has come to Python

#103

Post by owaislone »

sorentwo wrote: post #26
TkTech wrote: post #20

The Oban folks have done amazing, well-engineered work for years now - it's really the only option for Elixir. That said, I'm very confused at locking the process pool behind a pro subscription - this is basic functionality given CPython's architecture, not a nice-to-have. For $135/month on Oban Pro, they advertise: All Open Source Features Multi-Process Execution Workflows Global and Rate Limiting Unique Jobs Bulk O…

> It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues). We may well move some of those things to the OSS version, dependi…

Thanks. My two cents would be to not lock technical features behind a paywall. Lock "enterprise" features like encryption, FIPS, compliance reports, etc which make sense for a big corp. This would be far more palatable for someone small like my one-two person teams to adopt it and pay if we ever become big enough to care about enterprisey features.

Re: Oban, the job processing framework from Elixir, has come to Python

#104

Post by owaislone »

TkTech wrote: post #20

The Oban folks have done amazing, well-engineered work for years now - it's really the only option for Elixir. That said, I'm very confused at locking the process pool behind a pro subscription - this is basic functionality given CPython's architecture, not a nice-to-have. For $135/month on Oban Pro, they advertise: All Open Source Features Multi-Process Execution Workflows Global and Rate Limiting Unique Jobs Bulk O…

I'm sure you are aware but sharing anyway. Django 6.0 shipped an API called Django Tasks for background jobs so all Django code can implement portable, backend agnostic background jobs and swap backends on the fly but there are zero actual backends out there right now one can use in production. If you could add inbuilt Django Tasks support to Chancy or create a `django-chancy` package that bridged it, I think you'd see a lot of early adoption by Django projects.

Re: Oban, the job processing framework from Elixir, has come to Python

#105

Post by peheje »

simonw wrote: post #11

> Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back. This is such a key feature. Lots of people will tell you that you shouldn't use a relational database as a worker queue, but they inevitably miss out on how important transactions are for th…

This resonates so much. I spent years in an org watching "domain events" vanish into the ether because of the Dual Write Problem. We had these high-performance, sharded, distributed monsters that were "fast" on paper, but they couldn't guarantee a simple message would actually send after a record was saved.

Moving back to a rock-solid SQL-backed approach solved it overnight. But since there are no more "1% glitches," people have forgotten there was ever a fire. It’s a thankless win. The organization now thinks the system is "easy" and the "async purists" still lobby for a separate broker just to avoid "polluting" the DB. They’d rather trust complex, custom-built async logic than the most reliable part of their stack. (The transactional outbox pattern is essential, I just prefer mine backed by the same ACID guarantees as my data).

It’s tricky stuff. I'm an application dev, not a DB internalist, but I've realized that a week spent actually learning isolation levels and commit-ordering saves you a year of "distributed system" debugging. Even when teams layer an ORM like Entity Framework on top to "hide" the complexity, that SQL reality is still there. It’s not magic; it’s just ACID, and it’s been there the whole time.

Re: Oban, the job processing framework from Elixir, has come to Python

#106

Post by gethly »

tolien wrote: post #94
gethly wrote: post #87

When I hear Oban, the first thing that comes to mind is a great 14yo scotch :D

Same, although the town in general. I wondered if they addressed how they came up with the name, but don't see anything.

I've looked into it but have not found any mention or a clue.

Re: Oban, the job processing framework from Elixir, has come to Python

#107

Post by mozz100 »

owaislone wrote: post #91
kukkeliskuu wrote: post #82

Earlier quoted context omitted.

Most Django projects just need a basic way to execute timed and background tasks. Celery requires seperate containers or nodes, which complicates things unnecessarily. Django 6.0 luckily has tasks framework -- which is backported to earlier django versions has well, which can use the database. https://docs.djangoproject.com/en/6.0/topics/tasks/

Django 6's tasks framework is nice but so far it is only an API. It does not include an actual worker implementaiton. There is a django-tasks package which does a basic implementation but it is not prod ready. I tried it and it is very unreliable. Hopefully the community will come out with backends for it to plug celery, oban, rq etc.

Could you say a bit more about "it is very unreliable"? I'm considering using django-tasks with an rq backend [1] and would like to hear about your experiences. Did you find it dropping tasks, difficult to operate, etc.

[1] https://github.com/RealOrangeOne/django-tasks?tab=readme-ov-...

Post reply on HN