Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

From Python to Elixir Machine Learning

thestackcanary.com

Post reply on HN
Print view View original

41–50 of 89 posts

Re: From Python to Elixir Machine Learning

#41

Post by josevalim »

el_oni wrote: post #30
bongobingo1 wrote: post #22

Something not mentioned is Explorer ( https://hexdocs.pm/explorer/Explorer.html ) which is a very nice dataframe library backed by Rusts Polars https://www.pola.rs/ with great integrations for Livebook https://livebook.dev/ .

I found explorer quite frustrating. I've used polars in python and loved it, but I brought in some financial data and couldn't strip off "£" from the start of a string so I could go on to cast it to a number. As far as I could tell I would have to bring that data into elixir, do the text processing and put it back into explorer, which to me defeated the whole point of a dataframe library. I imagine it's good for prec…

José from the Nx/Explorer team here.

Feel free to open up an issue! We have been focused more on high-level features (such as integration with S3, Postgres, Snowflake, SQLite, etc) and therefore we are missing many functions that already exist on Polars. Good news is that it is very quick to add them, so just let us know. :)

Re: From Python to Elixir Machine Learning

#42

Post by catchnear4321 »

pythonaut_16 wrote: post #38
gv83 wrote: post #36

Earlier quoted context omitted.

still not easy at the most important thing of all: being approachable, instrumented and intuitive to people less dedicated to programming. I say this as someone that likes elixir, but after seeing it failing miserably at my org, I'm very skeptical it can be thrown around like a spring or node or django project. It needs real support from the org and requires module design skills that are not present in most random de…

Could you expand on what you mean by this? Module design doesn't seem any harder than class design in JS or Python. Do you mean the language is generally harder for non-developers? Or that Elixir is harder for JS/Python developers to pick up and write good code? Or something else? Writing well designed Elixir code does seem to require a fairly different approach from most common OO languages, at least at a surface le…

your last paragraph largely answers your questions.

it doesn’t take much to yeet some python scripts (and more) at the wall and get something to stick. SO, GPT, pick your poison, it’s painfully easy. solved problems as far as the eye can see, with a little glue or tape to hold it together.

elixir demands more of an investment. more than approximately zero is quite a bit if you already have momentum.

(elixir can absolutely be yeeted. not arguing otherwise.)

Re: From Python to Elixir Machine Learning

#43

Post by josevalim »

paulluuk wrote: post #40

Show me I can save a considerable amount of time training a 70b parameter huggingface model by switching to Elixir, then we can talk about me learning a new language.

The training costs and performance at that scale are mostly driven by hardware and I personally don't think language will make a large difference (I may be wrong).

Most of the Elixir Nx efforts are on inference, especially on how you can embed and scale it using concurrent and distributed patterns (see this post/video [1]). It may not be what you are looking for but we have more folks deploying than training models, so maybe they will find incentives to give Elixir a try. :)

[1]: https://news.livebook.dev/distributed2-machine-learning-note...

Re: From Python to Elixir Machine Learning

#44

Post by bongobingo1 »

el_oni wrote: post #30
bongobingo1 wrote: post #22

Something not mentioned is Explorer ( https://hexdocs.pm/explorer/Explorer.html ) which is a very nice dataframe library backed by Rusts Polars https://www.pola.rs/ with great integrations for Livebook https://livebook.dev/ .

I found explorer quite frustrating. I've used polars in python and loved it, but I brought in some financial data and couldn't strip off "£" from the start of a string so I could go on to cast it to a number. As far as I could tell I would have to bring that data into elixir, do the text processing and put it back into explorer, which to me defeated the whole point of a dataframe library. I imagine it's good for prec…

Seems that for that case you would need to use `mutate_with`, which as you say does pass data between systems.

My understanding is you can use any Series operations without penalty (ie. they get passed into some rust NIF call), https://hexdocs.pm/explorer/Explorer.Series.html#functions-s..., which does include whitespace trimming, but not arbitrary strings, but I imagine it wouldn't be too much of a jump to add arbitrary constant strings. Might just need to expose `str.slice` or `str.replace`.

The docs do imply that `mutate_with` operates lazily, so you only pay the transfer cost once per row, no matter how many mutations you're applying, but whether that's performant enough depends would be case by case.

Re: From Python to Elixir Machine Learning

#46

Post by throwawaymaths »

madspindel wrote: post #3

Instead of moving to Elixir I believe it makes more sense to wait/move to Mojo when it's ready: https://www.modular.com/mojo

Since it aims to be transparently compatible with python Mojo will very likely suffer from the worst part of python: packaging

Last week heard a story about an ML dev that would literally rebuild his system every week because python would break it

Re: From Python to Elixir Machine Learning

#47

Post by sodapopcan »

paulluuk wrote: post #40

Show me I can save a considerable amount of time training a 70b parameter huggingface model by switching to Elixir, then we can talk about me learning a new language.

Seems like you didn't read even the first line as this article is directed at people already using Elixir who are currently doing ML stuff with Python.

Re: From Python to Elixir Machine Learning

#48

Post by josevalim »

bongobingo1 wrote: post #44
el_oni wrote: post #30

Earlier quoted context omitted.

I found explorer quite frustrating. I've used polars in python and loved it, but I brought in some financial data and couldn't strip off "£" from the start of a string so I could go on to cast it to a number. As far as I could tell I would have to bring that data into elixir, do the text processing and put it back into explorer, which to me defeated the whole point of a dataframe library. I imagine it's good for prec…

Seems that for that case you would need to use `mutate_with`, which as you say does pass data between systems. My understanding is you can use any Series operations without penalty (ie. they get passed into some rust NIF call), https://hexdocs.pm/explorer/Explorer.Series.html#functions-s... , which does include whitespace trimming, but not arbitrary strings, but I imagine it wouldn't be too much of a jump to add arbi…

mutate_with is still lazy and therefore you can't transform it. If you need to use transform, then you need to do:

    new_column = Series.transform(df["column"], fn arg -> ... end)
    DF.put(df, "column", new_column)
which _is annoying_ since you are not supposed to use it. The correct way is to extend the Series API, which we will be very happy to!

Re: From Python to Elixir Machine Learning

#49

Post by throwawaymaths »

lots of negativity in here from people who have invested a lot of effort learning python, despite that there's also people who complain a lot about ML/DS in python. Is it an expression of sunk cost fallacy? If you don't support people who are trying to make a difference, nothing will change.

Re: From Python to Elixir Machine Learning

#50

Post by ricketycricket »

thibaut_barrere wrote: post #17

Don't miss the interesting book (which I just bought) in beta here: https://pragprog.com/titles/smelixir/machine-learning-in-eli...

Recommendation seconded. I'm only to chapter three due to time constraints, but it is well written and approachable for someone lacking ML experience.
Post reply on HN