Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oban, the job processing framework from Elixir, has come to Python

dimamik.com

Post reply on HN
Print view View original

41–50 of 107 posts

Re: Oban, the job processing framework from Elixir, has come to Python

#41

Post by mperham »

semiquaver wrote: post #5
mperham wrote: post #2

I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this! I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central se…

Isn’t it more accurate to say that they are both based on Resque?

Resque was the main inspiration, Sidekiq still provides compatibility with some of its APIs to this day.

https://github.com/sidekiq/sidekiq/blob/ba8b8fc8d81ac8f57a55...

Re: Oban, the job processing framework from Elixir, has come to Python

#42

Post by cpursley »

Oban is cool but I really like the idea of pgflow.dev, which is based on pgmq (rust) Postgres plugin doing the heavy lifting as it makes it language agnostic (all the important parts live in Postgres). I've started an Elixir adapter which really is just a DSL and poller, could do the same in Python, etc.

https://github.com/agoodway/pgflow

Re: Oban, the job processing framework from Elixir, has come to Python

#44

Post by mperham »

enraged_camel wrote: post #6
mperham wrote: post #2

I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this! I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central se…

Maybe you didn’t intend it this way, but your comment comes across as an attempt to co-opt the discussion to pitch your own thing. This is generally looked down upon here.

It was an off-the-cuff comment and probably not worded ideally but the intent was to discuss how Oban is branching off into a new direction for their business based on language-specific products while I went a different direction with Faktory. Since I came to the exact same fork in the road in 2017, I thought it was relevant and an interesting topic on evolving software products.

Re: Oban, the job processing framework from Elixir, has come to Python

#45

Post by qianli_cs »

sorentwo wrote: post #32
qianli_cs wrote: post #29

Thanks for sharing, interesting project! One thing that stood out to me is that some fairly core features are gated behind a Pro tier. For context, there are prior projects in this space that implement similar ideas fully in OSS, especially around Postgres-backed durable execution: 1. DBOS built durable workflows and queues on top of Postgres (disclaimer: I'm a co-founder of DBOS), with some recent discussions here:…

There are other projects that implement the ideas in OSS, but that's the same in Elixir. Not that we necessarily invented DAGs/workflows, but our durable implementation on the Elixir side predates DBOS by several years. We've considered it an add-on to what Oban offers, rather than the entire product. Having an entirely open source offering and selling support would be an absolute dream. Maybe we'll get there too.

That's fair, the idea itself isn't new. Workflows/durable execution have been around forever (same story in Elixir).

The differences are in the implementation and DX: the programming abstraction, how easy recovery/debugging is, and how it behaves once you're running a production cluster.

One thing that bit us early was versioning. In practice, you always end up with different workers running different code versions (rolling deploys, hotfixes, etc.). We spent a lot of time there and now support both workflow versioning and patching, so old executions can replay deterministically while still letting you evolve the code.

Curious how Oban handles versioning today?

Re: Oban, the job processing framework from Elixir, has come to Python

#46

Post by simonw »

nhumrich wrote: post #35
simonw wrote: post #11

> Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back. This is such a key feature. Lots of people will tell you that you shouldn't use a relational database as a worker queue, but they inevitably miss out on how important transactions are for th…

This is called the "transactional outbox pattern"!

Good name! Looks like SeatGeek use that naming convention here: https://chairnerd.seatgeek.com/transactional-outbox-pattern/

This looks like a good definition too: https://www.milanjovanovic.tech/blog/outbox-pattern-for-reli...

Re: Oban, the job processing framework from Elixir, has come to Python

#47

Post by sorentwo »

mperham wrote: post #34
ai_critic wrote: post #19

Earlier quoted context omitted.

"based on" is sorta a stretch here. Sidekiq is pretty bare bones compared to what Oban supports with workflows, crons, partitioning, dependent jobs, failure handling, and so forth.

By “based on” I don’t mean a shared codebase or features but rather Parker and I exchanged emails a decade ago to discuss business models and open source funding. He initially copied my Sidekiq OSS + Sidekiq Pro business model, with my blessing.

This is absolutely true (except we went OSS + Web initially, Pro came later). You were an inspiration, always helpful in discussion, and definitely paved the way for this business model.

Re: Oban, the job processing framework from Elixir, has come to Python

#48

Post by offbyone »

Ooof. I don't mind the OSS/pro feature gate for the most part, but I really don't love that "Pro version uses smarter heartbeats to track producer liveness."

There's a difference between QoL features and reliability functions; to me, at least, that means that I can't justify trying to adopt it in my OSS projects. It's too bad, too, because this looks otherwise fantastic.

Re: Oban, the job processing framework from Elixir, has come to Python

#49

Post by shepardrtc »

> Inaccurate rescues - jobs that are long-running might get rescued even if the producer is still alive. Pro version uses smarter heartbeats to track producer liveness.

So the non-paid version really can't be used for production unless you know for sure you'll have very short jobs?

Post reply on HN