Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

news.livebook.dev

Post reply on HN
Print view View original

21–30 of 56 posts

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#21

Post by peoplefromibiza »

adonig wrote: post #16
clircle wrote: post #4

Whoa, you know this is hacker news right? You don't get to call R elegant around these parts.

I remember when we first used R in a stochastic class. The professor (a mathematician) was in love with the language and the students (computer science) considered the language to be the PHP of science.

as a computer scientist and programming languages nerd, I think R is a much better language than Python (comparing the two only because Python is leading in the data science field)

I also believe that the tools available are superior, RStudio is very good IMO.

I wonder why R has such a bad reputation.

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#23

Post by peoplefromibiza »

Once again, Elixir community takes something and brings it to the next level.

In my opinion Jose Valim is the Linus Torvalds of programming languages, but differently from Linus not only he is a 100x engineer, he's also one of the humblest and kindest person I've ever met (don't get me wrong, I love Linus, but he can be "too honest" at times and comes out as harsh or rude).

From Elixir came Phoenix, from Phoenix came Liveview, from Liveview we got Livebook, iterating from good ideas to quality products like it's an easy task.

Can't wait to see what's the next trick in their sleeve.

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#24

Post by kyllo »

Elixir runs on the Erlang VM, which is designed for distributed multi-process concurrency, right? Why would this be advantageous for interactive data analysis work, which is typically done on a single node? I don't quite understand the use case, hoping someone can explain.

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#25

Post by peoplefromibiza »

kyllo wrote: post #24

Elixir runs on the Erlang VM, which is designed for distributed multi-process concurrency, right? Why would this be advantageous for interactive data analysis work, which is typically done on a single node? I don't quite understand the use case, hoping someone can explain.

Watch this

https://youtu.be/MSMyRBJAoSs

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#26

Post by kyllo »

peoplefromibiza wrote: post #21
adonig wrote: post #16

Earlier quoted context omitted.

I remember when we first used R in a stochastic class. The professor (a mathematician) was in love with the language and the students (computer science) considered the language to be the PHP of science.

as a computer scientist and programming languages nerd, I think R is a much better language than Python (comparing the two only because Python is leading in the data science field) I also believe that the tools available are superior, RStudio is very good IMO. I wonder why R has such a bad reputation.

A lot of programmers dislike R for the same reason they dislike PHP: weak typing.

If you can get over that, though, there's a lot to like about R. It's sort of like a hybrid between a Lisp and an array language.

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#27

Post by nestorD »

kyllo wrote: post #24

Elixir runs on the Erlang VM, which is designed for distributed multi-process concurrency, right? Why would this be advantageous for interactive data analysis work, which is typically done on a single node? I don't quite understand the use case, hoping someone can explain.

Uses cases that comes to mind are data distributed across nodes and things like distributed training of machine learning models (which is getting more and more focus as models get bigger).

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#28

Post by josevalim »

cjf4 wrote: post #5
hugobarauna wrote: post #2

Hi everyone, member of the Livebook team here. We’ve been investing a lot in making Elixir great for data exploration. Today we’re taking one step further in this journey by contributing to the Explorer library and integrating it with Livebook. Explorer is an Elixir dataframe library built on top of Polars (from Rust) and inspired by dplyr (from R). Its integration with Livebook (open-source code notebook for Elixir)…

Can you make a pitch to a Python/R user to give this a try? What you’ve built looks very nice and heard nothing but good things about elixir elsewhere, but would take a lot to leave those much more robust ecosystems. Do you hope to grow into that over time? Is there enough in terms of viz, statistical models, and ml to survive?

José from the Livebook team. I don't think I can make a pitch because I have limited Python/R experience to use as reference.

My suggestion is for you to give it a try for a day or two and see what you think. I am pretty sure you will find weak spots and I would be very happy to hear any feedback you may have. You can find my email on my GitHub profile (same username).

In general we have grown a lot since the Numerical Elixir effort started two years ago. Here are the main building blocks:

* Nx (https://github.com/elixir-nx/nx/tree/main/nx#readme): equivalent to Numpy, deeply inspired by JAX. Runs on both CPU and GPU via Google XLA (also used by JAX/Tensorflow) and supports tensor serving out of the box

* Axon (https://github.com/elixir-nx/axon): Nx-powered neural networks

* Bumblebee (https://github.com/elixir-nx/bumblebee): Equivalent to HuggingFace Transformers. We have implemented several models and that's what powers the Machine Learning integration in Livebook (see the announcement for more info: https://news.livebook.dev/announcing-bumblebee-gpt2-stable-d...)

* Explorer (https://github.com/elixir-nx/explorer): Series and DataFrames, as per this thread.

* Scholar (https://github.com/elixir-nx/scholar): Nx-based traditional Machine Learning. This one is the most recent effort of them all. We are treading the same path as scikit-learn but quite early on. However, because we are built on Nx, everything is derivable, GPU-ready, distributable, etc.

Regarding visualization, we have "smart cells" for VegaLite and MapLibre, similar to how we did "Data Transformations" in the video above. They help you get started with your visualizations and you can jump deep into the code if necessary.

I hope this helps!

Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R

#29

Post by [deleted] »

chubot wrote: post #8
clircle wrote: post #4

Whoa, you know this is hacker news right? You don't get to call R elegant around these parts.

R isn't elegant, but tidyverse is If you learn tidyverse, then you're going to cringe whenever you use Pandas or most things in the Python data science ecosystem https://www.rstudio.com/wp-content/uploads/2015/02/data-wran...

[deleted]
Post reply on HN