Extremely cool! This is the first time I'm learning about Livebook's smart cells, and seeing how easily you can toggle between the UI and the underlying chain of dataframe operations in code is pretty mind-blowing.
Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
31–40 of 56 posts
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#32Elixir runs on the Erlang VM, which is designed for distributed multi-process concurrency, right? Why would this be advantageous for interactive data analysis work, which is typically done on a single node? I don't quite understand the use case, hoping someone can explain.
Hi, maybe I can try to answer your question.
First, to makes ure we are on the same page, distribution in Erlang happens across nodes and concurrency happens within a single operating system process. Erlang calls its concurrency primitive processes (because they are also isolated and preemptive) but that can cause some confusion (hence this comment).
From now on, when I mention process, I mean the Erlang VM processes, and they are very lightweight and you can create millions of them.
I can think of a few different ways where concurrency can help interactive data analysis:
1. Livebook supports rich outputs where each output is a process. This means your notebook can communicate with outputs as it executes. For example, it is very easy for you train a neural network and push data to the graph as it comes. Or to process data and plot it as you go.
2. You can use concurrency to run several experiments at once within the same notebook. We support this in Livebook via "Branched sections". You can prepare the data and then start several branches/processes to digest the data in different ways without a need to start several notebooks.
3. It is also very easy to build applications where multiple users can collaborate and interact with it, which we showed yesterday: https://news.livebook.dev/build-and-deploy-a-whisper-chat-ap...
When it comes to distribution, it is quite similar to above, because the concurrency and distribution primitives in the Erlang VM are the same. Here is an example of how easy it is to take a ML model from concurrent to distributed: https://news.livebook.dev/distributed2-machine-learning-note...
Generally speaking, I think we should start from the opposite side: we should try to make everything concurrent by default and fallback to serial only when we cannot. Specially for data analysis, where moving data is expensive, we may end-up incurring a lot of overhead if the only form of concurrency is via the network or inter-process communication.
One last note, perhaps the most important bit of the Erlang VM machine for data analysis is that it favors a functional style. Livebook notebooks are strongly reproducible. I expand on this in this video: https://www.youtube.com/watch?v=EhSNXWkji6o
I hope this helps (and feel free to tell me if I missed the mark!).
PS: I know many of the videos above are machine learning related and that's because we have started our data journey only now. Although the principles should generally apply. Hopefully more data videos will come soon! :)
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#33The video overview says the journey in data is "just starting". That's exciting! Any ideas or vision for where it's going in the future?
Also, I noticed in the demo that installing polars was very fast! Was the video trimmed, or was it cached or something? I remember when I last tried out polars in Elixir a while ago, it had to build the rust library and everything and took like 10 minutes.
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#34The video overview says the journey in data is "just starting". That's exciting! Any ideas or vision for where it's going in the future? Also, I noticed in the demo that installing polars was very fast! Was the video trimmed, or was it cached or something? I remember when I last tried out polars in Elixir a while ago, it had to build the rust library and everything and took like 10 minutes.
We use the excellent Rustler Precompiled [1] library now so prebuilt binaries ship with the Elixir package. No Rust toolchain needed. :)
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#35Earlier quoted context omitted.
as a computer scientist and programming languages nerd, I think R is a much better language than Python (comparing the two only because Python is leading in the data science field) I also believe that the tools available are superior, RStudio is very good IMO. I wonder why R has such a bad reputation.
A lot of programmers dislike R for the same reason they dislike PHP: weak typing. If you can get over that, though, there's a lot to like about R. It's sort of like a hybrid between a Lisp and an array language.
Iirc the scoping is also very strange.
Certainly not as bad as mathematica, but bad nonetheless.
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#36The video overview says the journey in data is "just starting". That's exciting! Any ideas or vision for where it's going in the future? Also, I noticed in the demo that installing polars was very fast! Was the video trimmed, or was it cached or something? I remember when I last tried out polars in Elixir a while ago, it had to build the rust library and everything and took like 10 minutes.
Things that are in our roadmap in relation to our data vision:
* Data management within your notebook (https://github.com/livebook-dev/livebook/issues/1604) - we want you to be able to link files, urls, and object storages to your notebook and automatically manage/download it
* We want to make it easier to build visualizations (even easier than the current Chart smart cell) and also be able to filter a dataframe by selecting a visualization: https://github.com/livebook-dev/livebook/issues/1545
We have other ideas, such as making SQL a more prominent citizen in Livebook and be able to build a custom canvas as your work on your notebook, but those will likely take longer to realize.
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#37What is the rationale for using a programming language runtime designed for distributed computing for local data analysis?
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#38Earlier quoted context omitted.
as a computer scientist and programming languages nerd, I think R is a much better language than Python (comparing the two only because Python is leading in the data science field) I also believe that the tools available are superior, RStudio is very good IMO. I wonder why R has such a bad reputation.
A lot of programmers dislike R for the same reason they dislike PHP: weak typing. If you can get over that, though, there's a lot to like about R. It's sort of like a hybrid between a Lisp and an array language.
Weak typing? No, it's not. R is a strongly typed language.
> 1 + "string"
Error in 1 + "str" : non-numeric argument to binary operator
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#39Earlier quoted context omitted.
I think R is elegant, what's inelegant in it (other than the code lots of non-programmers write with it)?
The type system is so abhorrent, it makes me wonder if it's actually proper to call R a real programming language
what specifically is abhorrent about it?
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#40Earlier quoted context omitted.
as a computer scientist and programming languages nerd, I think R is a much better language than Python (comparing the two only because Python is leading in the data science field) I also believe that the tools available are superior, RStudio is very good IMO. I wonder why R has such a bad reputation.
A lot of programmers dislike R for the same reason they dislike PHP: weak typing. If you can get over that, though, there's a lot to like about R. It's sort of like a hybrid between a Lisp and an array language.
Javascript and C also have weak typing. And Python isn't statically typed, it's just that like Ruby, it doesn't allow implicit conversion, except with numbers.