Live data from Hacker News

Embedding Python in Elixir, it's fine

dashbit.co

11–20 of 66 posts

Re: Embedding Python in Elixir, it's fine

#11
post #8
post #7

Elixir has some features I wish Python had: - atoms - everything (or most things) is a macro, even def, etc. - pipes |>, and no, I don't want to write a "pipe" class in Python to use it like pipe(foo, bar, ...). 90% of the |> power comes from its 'flow' programming style. - true immutability - true parallelism and concurrency thanks to the supervision trees - hot code reloading (you recompile the app WHILE it's runni…

Mix is also so much better than anything python has to offer in terms of build/dependency tooling.

Jupyter might have fixed this now because it’s been a while since I used it, but Mix.install inline in Livebook (or any CLI script) is so much nicer than how installing Python dependencies in notebooks was last time I did that too.

Re: Embedding Python in Elixir, it's fine

#12
At first read this seems really promising. Getting into Elixir/Erlang ecosystem from Python has seemed too hard to take the time. And when there I wouldn't be able to leverage all the Python stuff I've learned. With Pythonx gradual learning seems now much more achievable.

It wasn't mentioned in the article, but there's older blog post on fly.io [1] about live book, GPUs, and their FLAME serverless pattern [2]. Since there seems to be some common ground between these companies I'm now hoping Pythonx support is coming to FLAME enabled Erlang VM. I'm just going off from the blog posts, and am probably using wrong terminology here.

For Python's GIL problem mentioned in the article I wonder if they have experimented with free threading [3].

[1] https://fly.io/blog/ai-gpu-clusters-from-your-laptop-liveboo...

[2] https://fly.io/blog/rethinking-serverless-with-flame/

[3] https://docs.python.org/3/howto/free-threading-python.html

Re: Embedding Python in Elixir, it's fine

#13
For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood.

For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's running on the same OS process as your Elixir app, you bypass the failure recovery that makes an Elixir/BEAM application so powerful.

Normally, an Elixir app has a supervision tree that can gracefully handle failures of its own BEAM processes (an internal concurrency unit -- kind like a synthetic OS process) and keep the rest of the app's processes running. That's one of the big selling points of languages like Elixir, Erlang, and Gleam that build upon the BEAM architecture.

Because it uses NIFs (natively-implemented functions), an unhandled exception in Pythonx would take down your whole OS process along with all other BEAM processes, making your supervision tree a bit worthless in that regard.

There are cases when NIFs are super helpful (for instance, Rustler is a popular NIF wrapper for Rust in Elixir), but you have to architect around the fact that it could take down the whole app. Using Ports (Erlang and Elixir's long-standing external execution handler) to run other native code like Python or Rust is less risky in this respect because the non-Elixir code it's still running in a separate OS process.

Re: Embedding Python in Elixir, it's fine

#16
post #13

For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood. For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's ru…

I hadn’t heard of gleam. Looks cool! I like working with elixir in a lot of ways but never was a Ruby guy, and I think I’d prefer the C-style syntax.

Re: Embedding Python in Elixir, it's fine

#17
post #9
post #7

Elixir has some features I wish Python had: - atoms - everything (or most things) is a macro, even def, etc. - pipes |>, and no, I don't want to write a "pipe" class in Python to use it like pipe(foo, bar, ...). 90% of the |> power comes from its 'flow' programming style. - true immutability - true parallelism and concurrency thanks to the supervision trees - hot code reloading (you recompile the app WHILE it's runni…

Coming from Erlang, I think macros are one of the things I'm ambivalent about in Elixir. There are a bunch of actual improvements besides just the syntax itself in Elixir, like string handling, but things like macros in Ecto ... not yet a fan of that.

> ..but things like macros in Ecto....

Well, The whole language itself is built on macros. The following series of articles certainly helped me stop worrying and love the macros..

https://www.theerlangelist.com/article/macros_1

Some interesting insights from the article: "Elixir itself is heavily powered by macros. Many constructs, such as defmodule, def, if, unless, and even defmacro[1] are actually macros...."

[1] https://github.com/elixir-lang/elixir/blob/v1.18.2/lib/elixi...

Re: Embedding Python in Elixir, it's fine

#18

At first read this seems really promising. Getting into Elixir/Erlang ecosystem from Python has seemed too hard to take the time. And when there I wouldn't be able to leverage all the Python stuff I've learned. With Pythonx gradual learning seems now much more achievable. It wasn't mentioned in the article, but there's older blog post on fly.io [1] about live book, GPUs, and their FLAME serverless pattern [2]. Since…

FLAME runs the same code base on another machine. FLAME with Pythonx should just work. FLAME is a set of nice abstractions on top of a completely regular Erlang VM.

Chris Grainger who pushed for the value of Python in Livebook has given at least two talks about the power and value of FLAME.

And of course Chris McCord (creator of Phoenix and FLAME) works at Fly and collaborates closely with Dashbit who do Livebook and all that.

These are some of the benefits of a cohesive ecosystem. Something I enjoy a lot in Elixir. All these efforts are aligned. There is nothing weird going on, no special work you need to do.

Re: Embedding Python in Elixir, it's fine

#19

Really glad to see this, Elixir has languished in the AI wars despite being a better fit than JavaScript and Python.

Forgive some ignorance on this; why is Elixir a better fit for AI than Python or JavaScript? I'm not disagreeing, I've just never heard that, I didn't think that Elixir had good linear algebra libraries like NumPy.

Re: Embedding Python in Elixir, it's fine

#20
As someone very involved in Elixir and who used to do a lot of Python this seems very practical for me. I'm actually even more interested in that Fine library for making C++ NIFs easy. That seems ridiculously valuable for removing hurdles to building library bindings.
Post reply on HN