Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embedding Python in Elixir, it's fine

dashbit.co

Post reply on HN
Print view View original

31–40 of 66 posts

Re: Embedding Python in Elixir, it's fine

#31

Post by tombert »

cpursley wrote: post #29
tombert wrote: post #19

Earlier quoted context omitted.

Forgive some ignorance on this; why is Elixir a better fit for AI than Python or JavaScript? I'm not disagreeing, I've just never heard that, I didn't think that Elixir had good linear algebra libraries like NumPy.

Sorry, I should have been more explicit: better for on the user facing implementation side (concurrency, streaming data, molding agent state, etc) vs the training side of things. If that makes sense.

Ah, fair enough. I've not done much with Elixir but I have done a fair amount with Erlang and you certainly don't need to sell me on how great it is for concurrency and distributed stuff.

Re: Embedding Python in Elixir, it's fine

#32

Post by chantepierre »

I love to see "well-known" people in the Elixir community endorsing and actively developing that kind of approach. Our VM and runtime does so much and is so well suited to orchestrating other languages and tech that it sometimes feels there's a standard track and an off-road track.

The difference between an off-road "sounds dangerous" idea and its safe execution is often only the quantity of work but our runtime encourages that. Here, it's a NIF so there's still a bit of risk, but it's always possible to spawn a separate BEAM instance and distribute yourself with it.

Toy example that illustrates it, first crashing with a NIF that is made to segfault :

  my_nif_app iex --name my_app@127.0.0.1 --cookie cookie -S mix
  iex(my_app@127.0.0.1)1> MyNifApp.crash
  [1]    97437 segmentation fault
In the second example, we have a "SafeNif" module that spawns another elixir node, connects to it, and runs the unsafe operation on it.

  my_nif_app iex --name my_app@127.0.0.1 --cookie cookie -S mix
  iex(my_app@127.0.0.1)1> MyNifApp.SafeNif.call(MyNifApp, :crash, [])
  Starting temporary node: safe_nif_4998973
  Starting node with: elixir --name safe_nif_4998973@127.0.0.1 --cookie :cookie --no-halt /tmp/safe_nif_4998973_init.exs
  Successfully connected to temporary node
  Calling MyNifApp.crash() on temporary node
  :error
  iex(my_app@127.0.0.1)2>
Thankfully Python, Zig and Rust should be good to go without that kind of dance :) .

Re: Embedding Python in Elixir, it's fine

#33

Post by ch4s3 »

djha-skin wrote: post #26

Elixir is just Lisp with a facelift[1], and lisps can be built on Python[2]. It stands to reason that an elixir-like can be built on Python too, so you could embed the Python runtime in Elixir but Elixir-likes are used to code for both. 1: https://wiki.alopex.li/ElixirForCynicalCurmudgeons 2: https://hylang.org/

The operating environment of the BEAM is what's great about elixir. Hy still has the GIL.

Re: Embedding Python in Elixir, it's fine

#34

Post by giancarlostoro »

chefandy wrote: post #16
bicx wrote: post #13

For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood. For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's ru…

I hadn’t heard of gleam. Looks cool! I like working with elixir in a lot of ways but never was a Ruby guy, and I think I’d prefer the C-style syntax.

I'm more of a Python and C# kind of guy, so Elixir never really hit the itch for me, but Gleam definitely does. One of these days I'll take a crack to see how I can use Gleam with Phoenix.

Re: Embedding Python in Elixir, it's fine

#35

Post by giancarlostoro »

bicx wrote: post #13

For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood. For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's ru…

Do any of them communicate with the BEAM? There used to be a Go based implementation of the BEAM that allowed you to drop-in with Go, I have to wonder if this could be done with Python so it doesn't interfere with what the BEAM is good that and lets Python code remain as-is.

Re: Embedding Python in Elixir, it's fine

#36

Post by alienthrowaway »

bicx wrote: post #13

For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood. For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's ru…

> Because it uses NIFs (natively-implemented functions), an unhandled exception in Pythonx would take down your whole OS process along with all other BEAM processes, making your supervision tree a bit worthless in that regard.

What's the Elixir equivalent if "Pythonic"? An architecture that allows a NIF to take down your entire supervision tree is the opposite of that, as it defeats a the stacks' philosophy.

The best practice for integrating Python into Elixir or Erlang would be to have an assigned genserver, or other supervision-tree element - responsible for hosting the Python NIF(s), and the design should allow for each branch or leaf of that tree to be killed/restarted safely, with no loss of state. BEAM message passing is cheap

Re: Embedding Python in Elixir, it's fine

#37

Post by sbrother »

shiandow wrote: post #27
behnamoh wrote: post #7

Elixir has some features I wish Python had: - atoms - everything (or most things) is a macro, even def, etc. - pipes |>, and no, I don't want to write a "pipe" class in Python to use it like pipe(foo, bar, ...). 90% of the |> power comes from its 'flow' programming style. - true immutability - true parallelism and concurrency thanks to the supervision trees - hot code reloading (you recompile the app WHILE it's runni…

You can abuse the '>>' notation in python for pipes (or you could use |, I suppose), but you'll have to deal with whitespace shenanigans. I'm also not entirely sure about the order of evaluation. And you'll need to do partial function application by hand if you want that (though it is possible to write a meta function for that). So one could write class Piped: def __init__(self, value): self.value = value def __or__(…

Apache Beam in Python does this, with code like

    counts = (
        lines
        | 'Split' >> (
            beam.FlatMap(
                lambda x: re.findall(r'[A-Za-z\']+', x)).with_output_types(str))
        | 'PairWithOne' >> beam.Map(lambda x: (x, 1))
        | 'GroupAndSum' >> beam.CombinePerKey(sum))
I'm not sure how I feel about it, other than the fact that I'd 100x rather write Beam pipelines in basically any other language. But that's about more than syntax.

Re: Embedding Python in Elixir, it's fine

#38

Post by bicx »

alienthrowaway wrote: post #36
bicx wrote: post #13

For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood. For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's ru…

> Because it uses NIFs (natively-implemented functions), an unhandled exception in Pythonx would take down your whole OS process along with all other BEAM processes, making your supervision tree a bit worthless in that regard. What's the Elixir equivalent if "Pythonic"? An architecture that allows a NIF to take down your entire supervision tree is the opposite of that, as it defeats a the stacks' philosophy. The best…

That's the thing though: a NIF execution isn't confined to the the BEAM process by its nature. From the Erlang docs:

> As a NIF library is dynamically linked into the emulator process, this is the fastest way of calling C-code from Erlang (alongside port drivers). Calling NIFs requires no context switches. But it is also the least safe, because a crash in a NIF brings the emulator down too. (https://www.erlang.org/doc/system/nif.html)

The emulator in this context is the BEAM VM that is running the whole application (including the supervisors).

Apparently Rustler has a way of running Rust NIFs but capturing Rust panics before they trickle down and crash the whole BEAM VM, but that seems like more of a Rust trick that Pythonx likely doesn't have.

The tl;dr is that NIFs are risky by default, and not really... Elixironic?

Re: Embedding Python in Elixir, it's fine

#39

Post by lawik »

giancarlostoro wrote: post #35
bicx wrote: post #13

For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood. For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's ru…

Do any of them communicate with the BEAM? There used to be a Go based implementation of the BEAM that allowed you to drop-in with Go, I have to wonder if this could be done with Python so it doesn't interfere with what the BEAM is good that and lets Python code remain as-is.

There are several libraries that allow a Python program to communicate with an Erlang program using Erlang Term Format and such.

This approach targets more performance-sensitive cases with stuff like passing data frames around and vectors/matrices that are costly to serialize/deserialize a lot of the time.

And it seems to make for a tighter integration.

Re: Embedding Python in Elixir, it's fine

#40

Post by ikety »

chefandy wrote: post #16
bicx wrote: post #13

For Livebook, this looks really cool. Love that it calls CPython directly via C++ NIFS in Elixir and returns Elixir-native data structures. That's a lot cleaner than interacting with Python in Elixir via Ports, which is essentially executing a `python` command under the hood. For production servers, Pythonx is a bit more risky (and the developers aren't claiming it's the right tool for this use case). Because it's ru…

I hadn’t heard of gleam. Looks cool! I like working with elixir in a lot of ways but never was a Ruby guy, and I think I’d prefer the C-style syntax.

My current favorite language, just no time to finish my gleam projects.
Post reply on HN