Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oban, the job processing framework from Elixir, has come to Python

dimamik.com

Post reply on HN
Print view View original

31–40 of 107 posts

Re: Oban, the job processing framework from Elixir, has come to Python

#31

Post by bdcravens »

semiquaver wrote: post #5
mperham wrote: post #2

I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this! I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central se…

Isn’t it more accurate to say that they are both based on Resque?

The API is very close, but architecturally it's different.

Additionally, delayed_job came before resque.

Re: Oban, the job processing framework from Elixir, has come to Python

#32

Post by sorentwo »

qianli_cs wrote: post #29

Thanks for sharing, interesting project! One thing that stood out to me is that some fairly core features are gated behind a Pro tier. For context, there are prior projects in this space that implement similar ideas fully in OSS, especially around Postgres-backed durable execution: 1. DBOS built durable workflows and queues on top of Postgres (disclaimer: I'm a co-founder of DBOS), with some recent discussions here:…

There are other projects that implement the ideas in OSS, but that's the same in Elixir. Not that we necessarily invented DAGs/workflows, but our durable implementation on the Elixir side predates DBOS by several years. We've considered it an add-on to what Oban offers, rather than the entire product.

Having an entirely open source offering and selling support would be an absolute dream. Maybe we'll get there too.

Re: Oban, the job processing framework from Elixir, has come to Python

#34

Post by mperham »

ai_critic wrote: post #19
mperham wrote: post #2

I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this! I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central se…

"based on" is sorta a stretch here. Sidekiq is pretty bare bones compared to what Oban supports with workflows, crons, partitioning, dependent jobs, failure handling, and so forth.

By “based on” I don’t mean a shared codebase or features but rather Parker and I exchanged emails a decade ago to discuss business models and open source funding. He initially copied my Sidekiq OSS + Sidekiq Pro business model, with my blessing.

Re: Oban, the job processing framework from Elixir, has come to Python

#35

Post by nhumrich »

simonw wrote: post #11

> Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back. This is such a key feature. Lots of people will tell you that you shouldn't use a relational database as a worker queue, but they inevitably miss out on how important transactions are for th…

This is called the "transactional outbox pattern"!

Re: Oban, the job processing framework from Elixir, has come to Python

#36

Post by TkTech »

sorentwo wrote: post #26
TkTech wrote: post #20

The Oban folks have done amazing, well-engineered work for years now - it's really the only option for Elixir. That said, I'm very confused at locking the process pool behind a pro subscription - this is basic functionality given CPython's architecture, not a nice-to-have. For $135/month on Oban Pro, they advertise: All Open Source Features Multi-Process Execution Workflows Global and Rate Limiting Unique Jobs Bulk O…

> It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues). We may well move some of those things to the OSS version, dependi…

> Support only options aren't going to cut it in our experience; but maybe that'll be different with Python.

That's totally fair, and I can only speak from the sidelines. I haven't had a chance to review the architecture - would it possibly make sense to swap from async as a free feature to the process pool, and make async a pro feature? This would help with adoption from other OSS projects, if that's a goal, as the transition from Celery would then be moving from a process pool to a process pool (for most users). The vast, vast majority of Python libraries are not async-friendly and most still rely on the GIL. On the other hand, Celery has absolutely no asyncio support at all, which sets the pro feature apart.

On the other hand, already released and as you said it's much harder to take a free feature and make it paid.

Thanks again for Oban - I used it for a project in Elixir and it was painless. Missing Oban was why I made Chancy in the first place.

Re: Oban, the job processing framework from Elixir, has come to Python

#37

Post by sorenone »

deeviant wrote: post #22

I can't imagine why you would want a job processing framework linked to a single thread, which make this seem like a paid-version-only product. What does it have over Celery?

The vast majority of tasks you use a job processing framework for are related to io bound side effects: sending emails, interacting with a database, making http calls, etc. Those are hardly impacted by the fact that it's a single thread. It works really well embedded in a small service.

You can also easily spawn as many processes running the cli as you like to get multi-core parallelism. It's just a smidge* little more overhead than the process pool backend in Pro.

Also, not an expert on Celery.

Re: Oban, the job processing framework from Elixir, has come to Python

#38

Post by sorentwo »

BowBun wrote: post #10
dfajgljsldkjag wrote: post #8

I have fixed many broken systems that used redis for small tasks. It is much better to put the jobs in the database we already have. This makes the code easier to manage and we have fewer things to worry about. I hope more teams start doing this to save time.

Traditional DBs are a poor fit for high-throughput job systems in my experience. The transactions alone around fetching/updating jobs is non-trivial and can dwarf regular data activity in your system. Especially for monoliths which Python and Ruby apps by and large still are. Personally I've migrated 3 apps _from_ DB-backed job queues _to_ Redis/other-backed systems with great success.

Transactions around fetching/updating aren't trivial, that's true. However, the work that you're doing _is_ regular activity because it's part of your application logic. That's data about the state of your overall system and it is extremely helpful for it to stay with the app (not to mention how nice it makes testing).

Regarding overall throughput, we've written about running one million jobs a minute [1] on a single queue, and there are numerous companies running hundreds of millions of jobs a day with oban/postgres.

[1]: https://oban.pro/articles/one-million-jobs-a-minute-with-oba...

Re: Oban, the job processing framework from Elixir, has come to Python

#39

Post by sorenone »

mperham wrote: post #34
ai_critic wrote: post #19

Earlier quoted context omitted.

"based on" is sorta a stretch here. Sidekiq is pretty bare bones compared to what Oban supports with workflows, crons, partitioning, dependent jobs, failure handling, and so forth.

By “based on” I don’t mean a shared codebase or features but rather Parker and I exchanged emails a decade ago to discuss business models and open source funding. He initially copied my Sidekiq OSS + Sidekiq Pro business model, with my blessing.

You got the beer. We got the pen. ;)

Re: Oban, the job processing framework from Elixir, has come to Python

#40

Post by sorentwo »

TkTech wrote: post #36
sorentwo wrote: post #26

Earlier quoted context omitted.

> It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues). We may well move some of those things to the OSS version, dependi…

> Support only options aren't going to cut it in our experience; but maybe that'll be different with Python. That's totally fair, and I can only speak from the sidelines. I haven't had a chance to review the architecture - would it possibly make sense to swap from async as a free feature to the process pool, and make async a pro feature? This would help with adoption from other OSS projects, if that's a goal, as the…

> The vast, vast majority of Python libraries are not async-friendly and most still rely on the GIL. On the other hand, Celery has absolutely no asyncio support at all, which sets the pro feature apart.

That's great advice. Wish we'd been in contact before =)

Post reply on HN