Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bridging Elixir and Python with Oban

oban.pro

Post reply on HN
Print view View original

41–50 of 57 posts

Re: Bridging Elixir and Python with Oban

#41

Post by Towaway69 »

OkayPhysicist wrote: post #36
Towaway69 wrote: post #26

Earlier quoted context omitted.

Reading that made me think how much that might be related to Elixir being very similar in syntax to Ruby. Do LLMs really differentiate between the two? Specific studies, as the one quoted, are a long way from original real world problems.

I suspect the biggest advantage Elixir has is the relative quality of the publicly available code. Approximately no one has Elixir as their first programming language, which keeps a lot of the absolute trash-tier code that we all make when first learning to program out of the training set. If you look at languages that are often people's first (Python, JavaScript, Java), only Java has an above average score. Of those…

Agree with the quality level but there are other languages where that is also the case: Erlang for example is probably one of those languages.

> Elixir's relationship to Ruby is pretty overstated

Perhaps I am actually am over thinking this. Elixir has probably diverged enough from Ruby (e.g. defmodule, pipe operators, :atom syntax) for LLMs to notice the difference between the two. But it does open the question, though, how does an LLM actually recognise the difference in code blocks in its training data.

There are probably many more programming languages where similarities exist.

Re: Bridging Elixir and Python with Oban

#42

Post by dnautics »

Towaway69 wrote: post #26
dnautics wrote: post #22

Earlier quoted context omitted.

Don't assume. Empirically, they are not. (This post Feb 2026 may change in future yadda yadda) See: autocodebench https://github.com/Tencent-Hunyuan/AutoCodeBenchmark/tree/ma...

Reading that made me think how much that might be related to Elixir being very similar in syntax to Ruby. Do LLMs really differentiate between the two? Specific studies, as the one quoted, are a long way from original real world problems.

Here are my thoughts:

https://m.youtube.com/watch?v=YZa5GqrZeao

Re: Bridging Elixir and Python with Oban

#43

Post by d4mi3n »

ricketycricket wrote: post #33
Towaway69 wrote: post #26

Earlier quoted context omitted.

Reading that made me think how much that might be related to Elixir being very similar in syntax to Ruby. Do LLMs really differentiate between the two? Specific studies, as the one quoted, are a long way from original real world problems.

Here are some thoughts on it from José Valim: https://dashbit.co/blog/why-elixir-best-language-for-ai LLMs absolutely understand and write good Elixir. I've done complex OTP and distributed work in tandem with Sonnet/Opus and they understand it well and happily keep up. All the Elixir constructs distinct from ruby are well applied: pipes, multiple function clauses, pattern matching, etc. I can say that anecdotally, C…

I suspect this is partly due to the quality of documentation for Elixir, Erlang, and BEAM. The OTP documentation has been around for a long time and has been excellently written. Erlang/Elixer doc gen outputs function signatures, arity, and both Elixir and Erlang handle concepts like function overloading in very explicit, well-defined ways.

Re: Bridging Elixir and Python with Oban

#44

Post by ch4s3 »

Towaway69 wrote: post #38
ch4s3 wrote: post #32

Earlier quoted context omitted.

Having written a lot of both languages, I'd be surprised if LLMs don't get tripped up on some of Ruby's semantics and weird stuff people do with monkey patching. I also find Ruby library documentation to be on average pretty poor.

> I also find Ruby library documentation to be on average pretty poor. That surprises me :) From my time doing Ruby (admittedly a few years back), I found libraries were very well documented and tested. But put into context of then (not now), documentation and testing weren't that popular amongst other programming languages. Ruby was definitely one of the drivers for the general adaption of TDD principles, for exampl…

I think they're often very well tested, but the documentation piece has always been lacking compared to Elixir.

I used to frequently find myself reading the source code of popular libraries or prying into them at runtime. There's also no central place or format for documentation in ruby. Yes rubydoc.info exists, but it's sort of an afterthought. Sidekiq uses a github wiki, Nokogiri has a dedicated site, Rails has a dedicated site, Ruby itself has yet another site. Some use RDoc, some don't. Or look at Devise https://rubydoc.info/github/heartcombo/devise/main/frames, there's simply nothing documented for most of the classes, and good luck finding in the docs where `before_action :authenticate_user!` comes from.

Re: Bridging Elixir and Python with Oban

#45

Post by bnchrch »

d4mi3n wrote: post #43
ricketycricket wrote: post #33

Earlier quoted context omitted.

Here are some thoughts on it from José Valim: https://dashbit.co/blog/why-elixir-best-language-for-ai LLMs absolutely understand and write good Elixir. I've done complex OTP and distributed work in tandem with Sonnet/Opus and they understand it well and happily keep up. All the Elixir constructs distinct from ruby are well applied: pipes, multiple function clauses, pattern matching, etc. I can say that anecdotally, C…

I suspect this is partly due to the quality of documentation for Elixir, Erlang, and BEAM. The OTP documentation has been around for a long time and has been excellently written. Erlang/Elixer doc gen outputs function signatures, arity, and both Elixir and Erlang handle concepts like function overloading in very explicit, well-defined ways.

Thats a large reason for sure!

I'd layer in a few more

* Largely stable and unchanged language through out its whole existance

* Authorship is largely senior engineers so the code you train on is high quality

* Relatively low number of abstractions in comparisson to other languages. Meaning there's less ways to do one thing.

* Functional Programming style pushes down hidden state, which lowers the complexity when understanding how a slice of a system works, and the likelyhood you introduce a bug

Re: Bridging Elixir and Python with Oban

#46

Post by ananthakumaran »

markstos wrote: post #25
ananthakumaran wrote: post #10

We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower…

Is this part of a web server or some other system where you could end up spawning N python processes instead of 1 at a time?

No, it's a background job. We can easily control the Python process count by controlling the job queue concurrency on the Elixir side.

Re: Bridging Elixir and Python with Oban

#47

Post by Kaliboy »

ananthakumaran wrote: post #10

We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower…

Honestly you saved yourself major possible headaches down the line with this approach.

At my work we run a fairly large webshop and have a ridiculous number of jobs running at all times. At this point most are running in Sidekiq, but a sizeable portion remain in Resque simply because it does just that, start a process.

Resque workers start by creating a fork, and that becomes the actual worker.

So when you allocate half your available RAM for the job, its all discarded and returned to the OS, which is FANTASTIC.

Sidekiq, and most job queues uses threads which is great, but all RAM allocated to the process stays allocated, and generally unused. Especially if you're using malloc it's especially bad. We used jemalloc for a while which helped since it allocates memory better for multithreaded applications, but easiest is to just create a process.

I don't know how memory intensive ML is, what generally screwed us over was image processing (ImageMagick and its many memory leaks) and... large CSV files. Yeah come to think of it, you made an excellent architectural choice.

Re: Bridging Elixir and Python with Oban

#48

Post by kzemek »

ananthakumaran wrote: post #10

We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower…

We also had a similar use case, so I built Snex[0] - specifically for Elixir-Python interop. Elixir-side spawns interpreters with Ports managed by GenServers, Python-side has a thin asyncio runtime to run arbitrary user code. Declarative environments (uv), optimized serde with language-specific objects (like `%MapSet{}` `set`), etc. Interpreters are meant to be long lived, so you pay for initialization once.

It's a very different approach than ex_cmd, as it's not really focused on the "streaming data" use case. Mine is a very command/reply oriented approach, though the commands can flow both ways (calling BEAM modules from Python). The assumption is that big data is passed around out of band; I may have to revisit that.

[0]: https://github.com/kzemek/snex

Re: Bridging Elixir and Python with Oban

#49

Post by Kaliboy »

This is a similar concept to Faktory, which uses a built in Redis server to manage shared job state.

You then implement workers in your language of choice and subscribe to queues.

Very interesting though, the article mentioned a few things I hadn't considered before like shared access to one database from multiple (different) apps.

I wonder how database schema state is handled in a case like that. And CI/CD.

Re: Bridging Elixir and Python with Oban

#50

Post by jbott »

ananthakumaran wrote: post #10

We have a similar use case. All Elixir code base, but need to use Python for ML libraries. We decided to use IPC. Elixir will spawn a process and communicate over stdio. https://github.com/akash-akya/ex_cmd makes it a breeze to stream stdin and stdout. This also has the added benefit of keeping the Python side completely stateless and keeping all the domain logic on the Elixir side. Spawning a process might be slower…

This might be of interest to others: Last night I stumbled across Hornbeam, a library in a similar vein from the author of Gunicorn that handles WSGI / ASGI apps as well as a specific wrapper for ML inference

https://erlangforums.com/t/hornbeam-wsgi-asgi-server-for-run... https://github.com/benoitc/hornbeam

Post reply on HN