Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Oban, the job processing framework from Elixir, has come to Python

dimamik.com

Post reply on HN
Print view View original

91–100 of 107 posts

Re: Oban, the job processing framework from Elixir, has come to Python

#91

Post by owaislone »

kukkeliskuu wrote: post #82
owaislone wrote: post #15

Earlier quoted context omitted.

I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?

Most Django projects just need a basic way to execute timed and background tasks. Celery requires seperate containers or nodes, which complicates things unnecessarily. Django 6.0 luckily has tasks framework -- which is backported to earlier django versions has well, which can use the database. https://docs.djangoproject.com/en/6.0/topics/tasks/

Django 6's tasks framework is nice but so far it is only an API. It does not include an actual worker implementaiton. There is a django-tasks package which does a basic implementation but it is not prod ready. I tried it and it is very unreliable. Hopefully the community will come out with backends for it to plug celery, oban, rq etc.

Re: Oban, the job processing framework from Elixir, has come to Python

#92

Post by nlnn »

mperham wrote: post #2

I wrote Sidekiq, which Oban is based on. Congratulations to Shannon and Parker on shipping this! I had to make this same decision years ago: do I focus on Ruby or do I bring Sidekiq to other languages? What I realized is that I couldn't be an expert in every language, Sidekiq.js, Sidekiq.py, etc. I decided to go a different direction and built Faktory[0] instead, which flips the architecture and provides a central se…

Faktory was a big influence/inpiration for Ocypod[0], a job queuing system I wrote a few years back (similarly language agnostic). Much appreciated for making it all open source.

[0]: https://github.com/davechallis/ocypod

Re: Oban, the job processing framework from Elixir, has come to Python

#93

Post by sorenone »

linter wrote: post #80

I recommend [Celery]( https://docs.celeryq.dev/en/stable/ ). Celery is a simple, flexible, and reliable distributed system to process vast amounts of messages, while providing operations with the tools required to maintain such a system. BTW its free.

How many people tried to pitch their own tool in this thread? We love that the article, discussions and the process has taught us more about what might need to be incorporated into the OSS version for Py. Once a feature is out there you can't retract it. It's v0.5 for this reason!

Free is good. We choose better. Free waits or competes with initial work priorities. We strive for zero open issues and real humans responding when you can't ship.

Re: Oban, the job processing framework from Elixir, has come to Python

#95

Post by xmcqdpt2 »

TkTech wrote: post #21
owaislone wrote: post #15

Earlier quoted context omitted.

I'm just coming back to web/API development Python after 7-8 years working on distributed systems in Go. I just built a Django+Celery MVP given what I knew from 2017 but I see a lot of "hate" towards Celery online these days. What issues have you run into with Celery? Has it gotten less reliable? harder to work with?

Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.

For an even lighter system than Celery, I'm a big fan of

https://python-rq.org/

It's super low on the dependencies and integrates nicely as a library into python applications. It's very bare bones.

Re: Oban, the job processing framework from Elixir, has come to Python

#96

Post by brianwawok »

hangonhn wrote: post #25
TkTech wrote: post #21

Earlier quoted context omitted.

Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.

Yeah that list right there. That's exactly it. We don't hate Celery at all. It's just a bit harder to get it to do certain things and requires a bit more coding and understanding of celery than what we want to invest time and effort in. Again, no hate towards Celery. It's not bad. We just want to see if there are better options out there.

But if you are too small for celery, it seems a hard sell to buy a premium message queue?

My top problem with my celery setup has always been visibility. AI and I spent and afternoon setting up a Prometheus / grafana server, and wiring celery into it. Has been a game changer. When things go crazy in prod, I can usually single it down to a specific task for a specific user. Has made my troubleshooting go from days to minutes. The actual queue and execute part has always been easy / worked well.

Re: Oban, the job processing framework from Elixir, has come to Python

#98

Post by xmcqdpt2 »

ammmir wrote: post #79
cpursley wrote: post #68

Earlier quoted context omitted.

Agree, and Claude Code does very well with Elixir despite TS/Python getting all the hype: https://youtu.be/iV1EcfZSdCM?si=KAJW26GVaBqZjR3M This helps with keeping it on track writing idiomatic elixir and using good patterns: https://skills.sh/agoodway/.claude/elixir-genius

> # WRONG: Elixir has no elsif How much context is eaten up by skills that rehash what a SOTA model should already know? Maybe token-wise, it's a wash: Elixir/OTP does a lot without third-party libs, which would require massive npm dependencies to achieve the same thing.

I think most of this trial and error "You are an experienced engineer" stuff probably hurts model performance. No one ever does comprehensive testing so eh, yolo.

https://github.com/agoodway/.claude/blob/main/skills/elixir-...

There are papers showing that models follow instructions less the more instructions they have. Now you think about how many instructions are embedded in that MD + the system prompt + likely a local AGENTS.md and at the end there is probably very little here that matters.

Re: Oban, the job processing framework from Elixir, has come to Python

#99

Post by cpursley »

xmcqdpt2 wrote: post #98
ammmir wrote: post #79

Earlier quoted context omitted.

> # WRONG: Elixir has no elsif How much context is eaten up by skills that rehash what a SOTA model should already know? Maybe token-wise, it's a wash: Elixir/OTP does a lot without third-party libs, which would require massive npm dependencies to achieve the same thing.

I think most of this trial and error "You are an experienced engineer" stuff probably hurts model performance. No one ever does comprehensive testing so eh, yolo. https://github.com/agoodway/.claude/blob/main/skills/elixir-... There are papers showing that models follow instructions less the more instructions they have. Now you think about how many instructions are embedded in that MD + the system prompt + likely a l…

Yeah, I honestly lean on the elixir agent one more over the full skill:

https://github.com/agoodway/.claude/blob/main/agents/elixir-...

Re: Oban, the job processing framework from Elixir, has come to Python

#100

Post by nxy7 »

offbyone wrote: post #48

Ooof. I don't mind the OSS/pro feature gate for the most part, but I really don't love that "Pro version uses smarter heartbeats to track producer liveness." There's a difference between QoL features and reliability functions; to me, at least, that means that I can't justify trying to adopt it in my OSS projects. It's too bad, too, because this looks otherwise fantastic.

This is what get's me about Oban too. It's very cool but it feels bad to have 'same thing but better' gated behind payment. I want the base thing to be the best version possible and extra features be paid (so that when I need them I might start paying). Feels like sometimes boundaries between Pro and Free are made in weird place almost as if someone went out of their way to make things behave differently (maybe it's just my inexperience with Oban though).
Post reply on HN