Bridging Elixir and Python with Oban
1–10 of 57 posts
Re: Bridging Elixir and Python with Oban
#2Very nice, Oban is great. I effectually found a similar approach with pgflow.dev (built around pgmq) - but the stateless deno "workers" are pretty unreliable and built an elixir worker (https://github.com/agoodway/pgflow) that can pick up and process jobs that were created by pgflow's supabase/typescript client. So maybe there's an opportunity also with Oban to have a TypeScript/Node client that can insert jobs that Elixir/Python Oban can pick up. Also, I wonder if another approach vs the python workers picking things up is to have elixir workers call/run python/lua, etc code or is that too limiting?
Re: Bridging Elixir and Python with Oban
#3Very nice, Oban is great. I effectually found a similar approach with pgflow.dev (built around pgmq) - but the stateless deno "workers" are pretty unreliable and built an elixir worker ( https://github.com/agoodway/pgflow ) that can pick up and process jobs that were created by pgflow's supabase/typescript client. So maybe there's an opportunity also with Oban to have a TypeScript/Node client that can insert jobs tha…
that's easy with https://hexdocs.pm/pythonx/Pythonx.html and https://hexdocs.pm/lua/Lua.html and works well too
Re: Bridging Elixir and Python with Oban
#4Very nice, Oban is great. I effectually found a similar approach with pgflow.dev (built around pgmq) - but the stateless deno "workers" are pretty unreliable and built an elixir worker ( https://github.com/agoodway/pgflow ) that can pick up and process jobs that were created by pgflow's supabase/typescript client. So maybe there's an opportunity also with Oban to have a TypeScript/Node client that can insert jobs tha…
btw, a lot of postgres envs are not going to have pgmq, so just use Oban and don't reinvent the wheel like I did ;)
Re: Bridging Elixir and Python with Oban
#5I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture.
This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your system should be organized.
Re: Bridging Elixir and Python with Oban
#6I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture. This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your…
What leads you to this conclusion
Re: Bridging Elixir and Python with Oban
#7I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture. This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your…
It’s almost as though choosing a single-threaded, GIL-encumbered interpreted scripting language as the primary interface to an ecosystem of extremely parallelized and concurrent high-performance hardware-dependent operations wasn’t quite the right move for our industry.
Re: Bridging Elixir and Python with Oban
#8I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture. This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your…
I disagree, using python for a web-server and something like celery for background work is a pretty common pattern.
My reading of this is it more or less allows you to use Postgres (which you're likely already using as your DB) for the task orchestration backend. And it comes with a cool UI.
Re: Bridging Elixir and Python with Oban
#9I feel like if you need to utilize a tool like this, odds are pretty good you may have picked the Wrong Tool For the Job, or, perhaps even worse, the wrong architecture. This is why it's so important to do lots of engineering before writing the first line of code on a project. It helps keep you from choosing a tool set or architecture out of preference and keeps you honest about the capabilities you need and how your…
Strange opinion. Plenty of apps have more than one language. I might end up using this.
Why? Because my app is built in Elixir and right now I’m also using a python app that is open source but I really just need a small part of the python app. I don’t wanna rewrite everything in Elixir because while it’s small I expect it to change over time (basically fetching a lot of data sources) and it will be pain to keep rewriting it when data collections needs to change (over a 100 different sources). Right now I run the python app as an api but it’s just so overkill and harder to manage vs just handling everything except the actually data collection in Elixir where I am already using Oban.
Re: Bridging Elixir and Python with Oban
#10We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower compared to enqueuing a job, but in our case the job usually takes long enough to make it irrelevant.