Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Embedding Python in Elixir, it's fine

dashbit.co

Post reply on HN
Print view View original

41–50 of 66 posts

Re: Embedding Python in Elixir, it's fine

#41

Post by chefandy »

giancarlostoro wrote: post #34
chefandy wrote: post #16

Earlier quoted context omitted.

I hadn’t heard of gleam. Looks cool! I like working with elixir in a lot of ways but never was a Ruby guy, and I think I’d prefer the C-style syntax.

I'm more of a Python and C# kind of guy, so Elixir never really hit the itch for me, but Gleam definitely does. One of these days I'll take a crack to see how I can use Gleam with Phoenix.

I’ve been mostly in Python, C# and C++ for the past decade or so but got into Elixir as my first functional language. Never got comfy with the syntax but dig how everything flows. Looking forward to digging into Gleam.

Re: Embedding Python in Elixir, it's fine

#43

Post by tommica »

chantepierre wrote: post #32

I love to see "well-known" people in the Elixir community endorsing and actively developing that kind of approach. Our VM and runtime does so much and is so well suited to orchestrating other languages and tech that it sometimes feels there's a standard track and an off-road track. The difference between an off-road "sounds dangerous" idea and its safe execution is often only the quantity of work but our runtime enco…

Its a neat way to do it - spin a temporary one, which can crash all it wants without affecting the other nodes. Fits like a glove to BEAM.

Re: Embedding Python in Elixir, it's fine

#44

Post by solid_fuel »

tombert wrote: post #19
cpursley wrote: post #14

Really glad to see this, Elixir has languished in the AI wars despite being a better fit than JavaScript and Python.

Forgive some ignorance on this; why is Elixir a better fit for AI than Python or JavaScript? I'm not disagreeing, I've just never heard that, I didn't think that Elixir had good linear algebra libraries like NumPy.

I’ve been actively using elixir for ML at work, and I would say it’s a solid choice.

The downside - unfortunately while bumblebee, Axon, and Nx are libraries that seem to have a fantastically engineered base most of the latest models don’t have native elixir implementations yet and making my own is a little beyond my skill still. So a lot of the models you can easily run are older.

But the advantages - easy long running processes, great multiprocessing support, solid error handling and recovery - all pair very well with AI systems.

For example, it’s very easy to make an application that grabs files, caches them locally, and runs ML tasks against them. You can use process monitoring and linking to manage the locally cached files, and there’s no runtime duration limit like you might hit in a serverless system like lambda. Interprocess messaging means you can easily run ML in a background task and stream results asynchronously to a user. Additionally, logs are automatically streamed to the parent process and it’s easy to tag logs with process metadata, so tracking what is going on in your application is dead simple.

That’s basically a whole stack for a live ML service with all the difficult infrastructure bits already taken care of.

Re: Embedding Python in Elixir, it's fine

#45

Post by neonsunset »

chefandy wrote: post #41
giancarlostoro wrote: post #34

Earlier quoted context omitted.

I'm more of a Python and C# kind of guy, so Elixir never really hit the itch for me, but Gleam definitely does. One of these days I'll take a crack to see how I can use Gleam with Phoenix.

I’ve been mostly in Python, C# and C++ for the past decade or so but got into Elixir as my first functional language. Never got comfy with the syntax but dig how everything flows. Looking forward to digging into Gleam.

If you liked Elixir but found it too "exotic" you may find F# enjoyable instead - it's a bit like Elixir but with a very powerful, gradually typed and fully inferred type system and has access to the rest of .NET. Excellent for scripting, data analysis and modeling of complex business domains. It's also very easy to integrate a new F# project into existing C# solution, and it ships with the SDK and is likely supported by all the tools you're already using. F# is also 5 to 10 times more CPU and memory-efficient.

(F# is one of the languages Elixir was influenced by and it is where Elixir got the pipe operator from)

Re: Embedding Python in Elixir, it's fine

#46

Post by conradfr »

davidw wrote: post #9
behnamoh wrote: post #7

Elixir has some features I wish Python had: - atoms - everything (or most things) is a macro, even def, etc. - pipes |>, and no, I don't want to write a "pipe" class in Python to use it like pipe(foo, bar, ...). 90% of the |> power comes from its 'flow' programming style. - true immutability - true parallelism and concurrency thanks to the supervision trees - hot code reloading (you recompile the app WHILE it's runni…

Coming from Erlang, I think macros are one of the things I'm ambivalent about in Elixir. There are a bunch of actual improvements besides just the syntax itself in Elixir, like string handling, but things like macros in Ecto ... not yet a fan of that.

In practice you don't really need to write macros if you don't make libraries.

Re: Embedding Python in Elixir, it's fine

#47

Post by solid_fuel »

jarpineh wrote: post #12

At first read this seems really promising. Getting into Elixir/Erlang ecosystem from Python has seemed too hard to take the time. And when there I wouldn't be able to leverage all the Python stuff I've learned. With Pythonx gradual learning seems now much more achievable. It wasn't mentioned in the article, but there's older blog post on fly.io [1] about live book, GPUs, and their FLAME serverless pattern [2]. Since…

Yeah, looks like it works fine, here's an example: https://pastebin.pl/view/a10aea3d

I'll add: FLAME is probably a great addition to pythonx. While a NIF can crash the node it is executed on, FLAME calls are executed on other nodes by default. So a crash here would only hard-crash processes on the same node (FLAME lets you group calls so that a flame node can have many being executed on it at any time).

Errors bubble back up to the calling process (and crash it by default but can be handled explicitly), so managing and retrying failed calls is easy.

Re: Embedding Python in Elixir, it's fine

#48

Post by kazinator »

pjmlp wrote: post #28
djha-skin wrote: post #26

Elixir is just Lisp with a facelift[1], and lisps can be built on Python[2]. It stands to reason that an elixir-like can be built on Python too, so you could embed the Python runtime in Elixir but Elixir-likes are used to code for both. 1: https://wiki.alopex.li/ElixirForCynicalCurmudgeons 2: https://hylang.org/

In a way Python is a bad Lisp, still looking forward that catches up in native code compilation and multiline lambdas. Could be better, but that is what mainstream gets.

We can sort of make a shitty multi-line lambda in Python by making a dummy function called progn, and using it, like this:

  >>> def progn(*args):
  ...   if args:
  ...     return args[-1]
  ...   else:
  ...     return None
  ...
  >>> fun = lambda x : progn(print('abc'),
  ...                        print(x),
  ...                        print('def'))
  >>>
  >>> fun(42)
  abc
  42
  def
:)

Re: Embedding Python in Elixir, it's fine

#49

Post by yellowapple »

Other commenters have already pointed out the safety implications of using NIFs for this. There are, however, other downsides worth considering:

- The Erlang VM scheduler can't preempt a NIF, so a long-running Python call risks hanging the VM. This is a non-issue for ports, since Python's running in a separate OS process. A NIF can mitigate this by spawning an OS thread and yielding until it finishes; ain't clear if that's what this library is doing.

- The article already mentions that the GIL prevents concurrent Python execution, but this would also be a non-issue for ports, since the Erlang caller would just spin up multiple Python interpreters. Does Python allow multiple interpreters per (OS) process, like e.g. Tcl does? If so, then that'd be a possible avenue for mitigating this issue while sticking with NIFs.

Re: Embedding Python in Elixir, it's fine

#50

Post by chefandy »

neonsunset wrote: post #45
chefandy wrote: post #41

Earlier quoted context omitted.

I’ve been mostly in Python, C# and C++ for the past decade or so but got into Elixir as my first functional language. Never got comfy with the syntax but dig how everything flows. Looking forward to digging into Gleam.

If you liked Elixir but found it too "exotic" you may find F# enjoyable instead - it's a bit like Elixir but with a very powerful, gradually typed and fully inferred type system and has access to the rest of .NET. Excellent for scripting, data analysis and modeling of complex business domains. It's also very easy to integrate a new F# project into existing C# solution, and it ships with the SDK and is likely supporte…

Been on the list for years — I’ll check it out.
Post reply on HN