Jose, you are prolific. Love it.
Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
51–56 of 56 posts
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#52Hi everyone, member of the Livebook team here. We’ve been investing a lot in making Elixir great for data exploration. Today we’re taking one step further in this journey by contributing to the Explorer library and integrating it with Livebook. Explorer is an Elixir dataframe library built on top of Polars (from Rust) and inspired by dplyr (from R). Its integration with Livebook (open-source code notebook for Elixir)…
Can you make a pitch to a Python/R user to give this a try? What you’ve built looks very nice and heard nothing but good things about elixir elsewhere, but would take a lot to leave those much more robust ecosystems. Do you hope to grow into that over time? Is there enough in terms of viz, statistical models, and ml to survive?
Jose's reply suggests the basics have Elixir equivalents. I can't really speak to that side but I can say the usability story is much much better.
The last time I gave Jupyter notebooks a go it was a full session of installing and updating various Python tools: pip, conda, jupyter then struggling with Python versions. You end up piecing together your own bespoke setup based on other people's outdated bespoke setups you find while searching for your error messages. Maybe that's better now, this was a few years ago. For Livebook it's "download the app and run it." Other options exist and are well documented and straight forward. I set up a livebook server on our k8s dev cluster with a pretty simple Deployment I wrote just from looking at the livebook README notes on docker. We've made livebooks that connect to the elixir app running in a different namespace on the cluster. Very cool.
Once you have Livebook going the `.livemd` file is both version control friendly AND very readable markdown file rather than the big json objects used in `.ipynb`.
For Livebook rebuilding cells is a lot more repeatable. It also does a good job of determining if a cell re-execution is necessary or not if a previous cell is modified which can save you a lot of time. Likewise the dependencies installed are captured at the top so I've never had a problem when sharing a livebook. The other person always gets the same results that I had. I don't remember how it worked for Jupyter but it's really cool to collaborate with someone by both going to the same notebook session. It's like working on the same Google Doc but you are writing and executing code.
Now with the Publish functionality I can see using a livebook to throw together some functionality and share it with non-technical users in your org, while having it backed up to git for posterity.
I avoided Smart Cells for a while because I didn't like the "magic-ness" of the UI hiding what the code was doing, but as Jose has shown in the launch videos this week you can easily see the code they are backed with and replace the cell with the code if you want to take full control. Maybe it was always like that but I didn't realize it at first. They really make setting up stuff very easy without limiting you later on.
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#53Earlier quoted context omitted.
Things that are in our roadmap in relation to our data vision: * Data management within your notebook ( https://github.com/livebook-dev/livebook/issues/1604 ) - we want you to be able to link files, urls, and object storages to your notebook and automatically manage/download it * We want to make it easier to build visualizations (even easier than the current Chart smart cell) and also be able to filter a dataframe by…
A vote here for a SQL cell. I want folks to use Livebook and Explorer more, but a very easy win for data folks who are not familiar with Ecto and are mostly writing complex select statements would be a sql code block that can easily reference a connection. That would let people who are getting into Elixir for data work run a query, get an Explorer.DataFrame, and interact further that way.
[deleted]
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#54Earlier quoted context omitted.
Things that are in our roadmap in relation to our data vision: * Data management within your notebook ( https://github.com/livebook-dev/livebook/issues/1604 ) - we want you to be able to link files, urls, and object storages to your notebook and automatically manage/download it * We want to make it easier to build visualizations (even easier than the current Chart smart cell) and also be able to filter a dataframe by…
A vote here for a SQL cell. I want folks to use Livebook and Explorer more, but a very easy win for data folks who are not familiar with Ecto and are mostly writing complex select statements would be a sql code block that can easily reference a connection. That would let people who are getting into Elixir for data work run a query, get an Explorer.DataFrame, and interact further that way.
Good feedback, thanks.
Livebook already has a SQL Smart cell (https://livebook.dev/integrations/sql). It doesn't integrate with Explorer yet, but it's already possible to reference a database connection, run a SQL query, and visualize the results in a table.
Here's a video showing how to do that https://www.youtube.com/watch?v=F98OWdigCjY
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#55Whoa, you know this is hacker news right? You don't get to call R elegant around these parts.
R isn't elegant, but tidyverse is If you learn tidyverse, then you're going to cringe whenever you use Pandas or most things in the Python data science ecosystem https://www.rstudio.com/wp-content/uploads/2015/02/data-wran...
Oh, I do cringe at Pandas.
Re: Data wrangling in Elixir with Explorer, the power of Rust, the elegance of R
#56Earlier quoted context omitted.
I remember when we first used R in a stochastic class. The professor (a mathematician) was in love with the language and the students (computer science) considered the language to be the PHP of science.
as a computer scientist and programming languages nerd, I think R is a much better language than Python (comparing the two only because Python is leading in the data science field) I also believe that the tools available are superior, RStudio is very good IMO. I wonder why R has such a bad reputation.
Because it’s built around a very specialized set of needs (data manipulation, visualization, and statistical modeling), and it is essentially best in class at it, but it has quirks as a result. Anyone coming to R from a background in another language will feel those quirks intensely and assume it’s bad.