Live data from Hacker News

From Python to Elixir Machine Learning

thestackcanary.com

51–60 of 89 posts

Re: From Python to Elixir Machine Learning

#51
post #36

Earlier quoted context omitted.

I come from Ruby but the reactions can be similar, happy to give my data point. The thing is Elixir is really good at an increasing number of things. If you need to write a HTTP proxy in the middle of your application, since Elixir processes & incoming HTTP workers are cheap, you do not need to go evented: it just works. If you need to have reactive web apps with automated changes pushed to the client, it's the same:…

still not easy at the most important thing of all: being approachable, instrumented and intuitive to people less dedicated to programming. I say this as someone that likes elixir, but after seeing it failing miserably at my org, I'm very skeptical it can be thrown around like a spring or node or django project. It needs real support from the org and requires module design skills that are not present in most random de…

On the other hand, it may filter less dedicated people from showing up at your door.

Re: From Python to Elixir Machine Learning

#52

I use Elixir for a long time and wouldn't recommend it for ML over Python. Yes, maybe Elixir has some advantages and solutions for pain points because of VM architecture and other subjective reasons but doesn't come close to what Python offers in terms of tooling, support, community. Also, if you juggle more complex data in Elixir, it's something that you need to get used to coming from Python, it will be much much d…

I can no longer edit parent post so to further elaborate on tooling: even if Elixir was comparable to Python (let's assume it's true as of today), there is a lot going on in AI, so that won't necessarily be true tomorrow. There is no doubt Elixir will always be slower to catch up to recent development because of smaller community. This also applies to how quickly issues will be resolved as they appear. It's likely OK if you're toying with a hobby ML project on your own, but not something you'd want in a startup with stakeholders who have expectations in regards to timeline, and so on...

I like Elixir for web development otherwise, it is a much more stable domain so above doesn't apply (although I've seen some claim otherwise, which is telling how much more of an issue it would be for niche ML use case).

I'd be very happy to be proven wrong by some case studies of how companies leveraged Elixir in real ML projects and concluded it is superior to Python.

Re: From Python to Elixir Machine Learning

#53
post #36

Earlier quoted context omitted.

still not easy at the most important thing of all: being approachable, instrumented and intuitive to people less dedicated to programming. I say this as someone that likes elixir, but after seeing it failing miserably at my org, I'm very skeptical it can be thrown around like a spring or node or django project. It needs real support from the org and requires module design skills that are not present in most random de…

Could you expand on what you mean by this? Module design doesn't seem any harder than class design in JS or Python. Do you mean the language is generally harder for non-developers? Or that Elixir is harder for JS/Python developers to pick up and write good code? Or something else? Writing well designed Elixir code does seem to require a fairly different approach from most common OO languages, at least at a surface le…

sure thing! I'll try to make points related to what I observed and experienced in my org, starting with one huge huge huge preamble which is: I work in an average org with average joe devs. an average joe dev to me is someone who "just wants a job" and is not very interested in furthering its own professional development or learning new things unless trained & forced by the job. it's perfectly fine to be an average dev, I understand I'm sounding like a snub but the difference is real, exists, it's tangible as I touch it every day.

having said that, designing a proper elixir module (which basically is a bunch of functions that operates on a certain data structure usually represented by map of a certain shape) carries a certain level of fundamental understanding of the operation you're doing, which in my experience is one of the hardest things to get correctly.

It also requires a different way of exploring code as you can't do the familiar `.` and see what happens, you can't just do `price.toCentsValue` but need to do `price |> Price.toCentsValue` and so you need to know the existence of the `Price` module, which might not exist and be buried in `Cart` or as an helper in some controller because you did not understand clearly the domain and the modules responsibilities. Attaching behaviour to data is powerful, explorable, and most people are used to it, even if it's the wrong place in principle, with modules this is flipped, it's now data that must thread through operations, and it's not super easy to grasp.

Also, tooling is not that good (dialyzer sucks, intellij plugin not that good, vscode lsp good but still not a proper experience from people coming from c#/java, type annotations are not that readable...), pattern matching and destructuring on fn arguments confuses people and it's not super easy to read, and a million other papercuts related to tooling and syntax.

We don't have many elixir codebases (let's say around 15-25%) and I've seen incessant whining about "we don't want to maintain elixir" simply because the majority of people cannot be bothered to learn another mountain of quirks and papercuts (every lang has them) plus also losing the familiar way of working they already have, and having to remember that for the spot ticket that appears once in a blue moon on jira for elixir. That's why I think elixir needs extra support from the organization, basically in mandating it to be the primary language, teaching people proper design + proper code navigation and structure techniques, etc.

I hope I've been clear in my long winded ramblings; and I still wish a great future for elixir, so it becomes more approachable in average places like mine

Re: From Python to Elixir Machine Learning

#54
post #51
post #36

Earlier quoted context omitted.

still not easy at the most important thing of all: being approachable, instrumented and intuitive to people less dedicated to programming. I say this as someone that likes elixir, but after seeing it failing miserably at my org, I'm very skeptical it can be thrown around like a spring or node or django project. It needs real support from the org and requires module design skills that are not present in most random de…

On the other hand, it may filter less dedicated people from showing up at your door.

this has been a point of contention with numerous hr departments that needs to hire fast "cuz investors!!11". this is the reality of life sadly

Re: From Python to Elixir Machine Learning

#55
post #32

Earlier quoted context omitted.

I come from Ruby but the reactions can be similar, happy to give my data point. The thing is Elixir is really good at an increasing number of things. If you need to write a HTTP proxy in the middle of your application, since Elixir processes & incoming HTTP workers are cheap, you do not need to go evented: it just works. If you need to have reactive web apps with automated changes pushed to the client, it's the same:…

And nerves makes it easy to do IoT stuff / almost-embedded

add rustler and you can even do some systems stuff! (We use it at work)

Re: From Python to Elixir Machine Learning

#56

lots of negativity in here from people who have invested a lot of effort learning python, despite that there's also people who complain a lot about ML/DS in python. Is it an expression of sunk cost fallacy? If you don't support people who are trying to make a difference, nothing will change.

It's a weird mental block too - learning a language like Elixir, being so well designed and documented, is as comfortable as it gets.

Devs we hire without direct Elixir experience pick it up really quick (within a couple weeks). The energy needed to "get good" with Elixir is really not much considering it provides veritable super powers on the backend and introduces a whole category of concurrency concepts that are not easy to grasp elsewhere.

Re: From Python to Elixir Machine Learning

#57

Instead of moving to Elixir I believe it makes more sense to wait/move to Mojo when it's ready: https://www.modular.com/mojo

Mojo is pretty cool but its a difficult road to take - especially to implement the breadth of accrued functionality of a 20 year old language like Python.

Elixir & Python are not an apples to apples comparison - there are fundamental differences in the programming model (functional, immutability, etc) and runtime (preemptive scheduling + OTP) that is the reason it has distinct advantages not available without heavy cost trade-offs elsewhere.

Either way once Mojo is production ready Elixir will be able to use it as well like it does Rust, Zig, or Python.

Re: From Python to Elixir Machine Learning

#58
post #30

Earlier quoted context omitted.

I found explorer quite frustrating. I've used polars in python and loved it, but I brought in some financial data and couldn't strip off "£" from the start of a string so I could go on to cast it to a number. As far as I could tell I would have to bring that data into elixir, do the text processing and put it back into explorer, which to me defeated the whole point of a dataframe library. I imagine it's good for prec…

José from the Nx/Explorer team here. Feel free to open up an issue! We have been focused more on high-level features (such as integration with S3, Postgres, Snowflake, SQLite, etc) and therefore we are missing many functions that already exist on Polars. Good news is that it is very quick to add them, so just let us know. :)

Thanks José!

Yeah I noticed that, it looks like the numeric manipulations are well represented, less so for strings.

I'll dig out the the code later on and get an issue raised.

Re: From Python to Elixir Machine Learning

#59
post #30

Earlier quoted context omitted.

I found explorer quite frustrating. I've used polars in python and loved it, but I brought in some financial data and couldn't strip off "£" from the start of a string so I could go on to cast it to a number. As far as I could tell I would have to bring that data into elixir, do the text processing and put it back into explorer, which to me defeated the whole point of a dataframe library. I imagine it's good for prec…

You can use str.slice or str.extract to clean the data: https://pola-rs.github.io/polars/py-polars/html/reference/ex...

Yes for polars in python. This was an issue I had with explorer in elixir. There is no elixir binding to the str.slice or str.extract functions yet

Re: From Python to Elixir Machine Learning

#60

I wish there wouldn’t be such a song and dance about “moving away from Python”. There’s nothing wrong with creating ML tools in Elixir, but it’s always Python is slow, Python has no concurrency support, blah blah

Python is slow and concurrency is not great.
Post reply on HN