Live data from Hacker News

Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

news.livebook.dev

41–50 of 56 posts

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#41

Earlier quoted context omitted.

I think R is elegant, what's inelegant in it (other than the code lots of non-programmers write with it)?

The type system is so abhorrent, it makes me wonder if it's actually proper to call R a real programming language

And from a different point of view, real programming languages have built-in vectorization and 1-based arrays ;)

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#42

Once again, Elixir community takes something and brings it to the next level. In my opinion Jose Valim is the Linus Torvalds of programming languages, but differently from Linus not only he is a 100x engineer, he's also one of the humblest and kindest person I've ever met (don't get me wrong, I love Linus, but he can be "too honest" at times and comes out as harsh or rude). From Elixir came Phoenix, from Phoenix came…

IIRC he was behind the very popular Rails authentication gem Devise, as well. Really unbelievable how much of a boon he’s been to the open source community.

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#43
post #37

What is the rationale for using a programming language runtime designed for distributed computing for local data analysis?

Languages and runtimes often grow beyond their original scope. And since the introduction of Dirty NIFs to the Erlang VM five years (or so) ago, integrating with native code (which is what powers a lot of data analysis and machine learning tools in high-level languages) has become a real possibility. There is a similar-ish discussion here: https://news.ycombinator.com/item?id=35572128

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#44

Earlier quoted context omitted.

The type system is so abhorrent, it makes me wonder if it's actually proper to call R a real programming language

what specifically is abhorrent about it?

The coercion always gets on my nerves, JavaScript gets a bad rep but R is pretty damn warty too; weird ass data types ('ordered factor', anyone) that just seem so very far away from design choices in other languages without being particularly ergonomic or aesthetically appealing

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#45
post #5

Earlier quoted context omitted.

Can you make a pitch to a Python/R user to give this a try? What you’ve built looks very nice and heard nothing but good things about elixir elsewhere, but would take a lot to leave those much more robust ecosystems. Do you hope to grow into that over time? Is there enough in terms of viz, statistical models, and ml to survive?

José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference. My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username). In general we have grown a lot since the Numeric…

Edit: I wrote down an introduction to all of these on our elixir-nx organization page on GitHub: https://github.com/elixir-nx

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#46

Once again, Elixir community takes something and brings it to the next level. In my opinion Jose Valim is the Linus Torvalds of programming languages, but differently from Linus not only he is a 100x engineer, he's also one of the humblest and kindest person I've ever met (don't get me wrong, I love Linus, but he can be "too honest" at times and comes out as harsh or rude). From Elixir came Phoenix, from Phoenix came…

The dude is superhuman. An absolute machine in terms of programming output. Very engaged with the community. And extremely patient with people who have wrong opinions :)

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#48

Earlier quoted context omitted.

what specifically is abhorrent about it?

The coercion always gets on my nerves, JavaScript gets a bad rep but R is pretty damn warty too; weird ass data types ('ordered factor', anyone) that just seem so very far away from design choices in other languages without being particularly ergonomic or aesthetically appealing

The data types make sense to statisticians. Ordered factors are great when you need to fit, say, an ordered logit regression model.

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#49

Earlier quoted context omitted.

what specifically is abhorrent about it?

The coercion always gets on my nerves, JavaScript gets a bad rep but R is pretty damn warty too; weird ass data types ('ordered factor', anyone) that just seem so very far away from design choices in other languages without being particularly ergonomic or aesthetically appealing

Which coercion specifically are you talking about? Could you give an example?

Weird data types: R had been designed for and by statisticians with their specific needs in mind, which indeed could look weird to regular people.

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#50

The video overview says the journey in data is "just starting". That's exciting! Any ideas or vision for where it's going in the future? Also, I noticed in the demo that installing polars was very fast! Was the video trimmed, or was it cached or something? I remember when I last tried out polars in Elixir a while ago, it had to build the rust library and everything and took like 10 minutes.

Things that are in our roadmap in relation to our data vision: * Data management within your notebook ( https://github.com/livebook-dev/livebook/issues/1604 ) - we want you to be able to link files, urls, and object storages to your notebook and automatically manage/download it * We want to make it easier to build visualizations (even easier than the current Chart smart cell) and also be able to filter a dataframe by…

A vote here for a SQL cell. I want folks to use Livebook and Explorer more, but a very easy win for data folks who are not familiar with Ecto and are mostly writing complex select statements would be a sql code block that can easily reference a connection.

That would let people who are getting into Elixir for data work run a query, get an Explorer.DataFrame, and interact further that way.

Post reply on HN