51–60 of 107 posts
Earlier quoted context omitted.
"based on" is sorta a stretch here. Sidekiq is pretty bare bones compared to what Oban supports with workflows, crons, partitioning, dependent jobs, failure handling, and so forth.
By “based on” I don’t mean a shared codebase or features but rather Parker and I exchanged emails a decade ago to discuss business models and open source funding. He initially copied my Sidekiq OSS + Sidekiq Pro business model, with my blessing.
> Oban allows you to insert and process jobs using only your database. You can insert the job to send a confirmation email in the same database transaction where you create the user. If one thing fails, everything is rolled back. This is such a key feature. Lots of people will tell you that you shouldn't use a relational database as a worker queue, but they inevitably miss out on how important transactions are for th…
> Inaccurate rescues - jobs that are long-running might get rescued even if the producer is still alive. Pro version uses smarter heartbeats to track producer liveness. So the non-paid version really can't be used for production unless you know for sure you'll have very short jobs?
Ooof. I don't mind the OSS/pro feature gate for the most part, but I really don't love that "Pro version uses smarter heartbeats to track producer liveness." There's a difference between QoL features and reliability functions; to me, at least, that means that I can't justify trying to adopt it in my OSS projects. It's too bad, too, because this looks otherwise fantastic.
Single-threaded asyncio execution - concurrent but not truly parallel, so CPU-bound jobs block the event loop.
This makes it not even worth trying. Celery's interface kind of sucks, but I'm used to it already, and I can get infinitely parallel expanding vertically and horizontally for as long as I can afford the resources.
I also don't particularly like ayncio, and if I'm using a job queue wouldn't expect to need it.
Edit: I looked into it a bit more, and it seems we can launch multiple worker nodes, which doesn't seem as bad as what I originally thought
Ooof. I don't mind the OSS/pro feature gate for the most part, but I really don't love that "Pro version uses smarter heartbeats to track producer liveness." There's a difference between QoL features and reliability functions; to me, at least, that means that I can't justify trying to adopt it in my OSS projects. It's too bad, too, because this looks otherwise fantastic.
With a typical Redis or RabbitMQ backed durable queue you’re not guaranteed to get the job back at all after an unexpected shutdown. That quote is also a little incorrect—producer liveness is tracked the same way, it’s purely how “orphaned” jobs are rescued that is different.
I can't imagine why you would want a job processing framework linked to a single thread, which make this seem like a paid-version-only product. What does it have over Celery?
The vast majority of tasks you use a job processing framework for are related to io bound side effects: sending emails, interacting with a database, making http calls, etc. Those are hardly impacted by the fact that it's a single thread. It works really well embedded in a small service. You can also easily spawn as many processes running the cli as you like to get multi-core parallelism. It's just a smidge* little mo…
I also use celery when I have a process a user kicked off by clicking a button and they're watching the progress bar in the gui. One process might have 50 tasks, or one really long task.
Edit: I looked into it a bit more, and it seems we can launch multiple worker nodes, which doesn't seem as bad as what I originally thought
Earlier quoted context omitted.
With a typical Redis or RabbitMQ backed durable queue you’re not guaranteed to get the job back at all after an unexpected shutdown. That quote is also a little incorrect—producer liveness is tracked the same way, it’s purely how “orphaned” jobs are rescued that is different.
"jobs that are long-running might get rescued even if the producer is still alive" indicates otherwise. It suggests that jobs that are in progress may be double-scheduled. That's a feature that I think shouldn't be gated behind a monthly pro subscription; my unpaid OSS projects don't justify it.
This feels like core functionality is locked away, and the opensource part is nothing more than a shareware, or demo/learning version.
Edit: I looked into it a bit more, and it seems we can launch multiple worker nodes, which doesn't seem as bad as what I originally thought