Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
datarevenue.com
Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
1–10 of 34 posts
Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
#2It would be interesting to see koalas compared as well
Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
#3It would be interesting to see koalas compared as well
Koalas is the Pandas API on top of Apache Spark for anyone that's interested: https://github.com/databricks/koalas
It works similar to PySpark and is scalable to massive datasets (hundreds of terabytes). Koalas is probably the best bet if you're working on a massive dataset and want the Pandas API. Or you can simply use PySpark which has a cleaner interface.
Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
#4Oh, I wrote this :) I submitted it last week but it didn't get much attention then.
Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them.
If you know more than I do and I made any mistakes, let me know and I'll get them corrected.
Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
#5Oh, I wrote this :) I submitted it last week but it didn't get much attention then. Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them. If you know more than I do and I made any mistakes, let me know and I'll get them corrected.
Do you also have experience working with SQL databases? If so, how do they compare to Pandas in terms of performance? (with or without these extensions)
Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
#6Articles like these are interesting, but what surprises me is that they rarely set up a holistic use case, so most debates imagine how long it would take an expert user to use each tool. But time constraints (eg spent coding) separates expert from novice performance in many domains.
FWIW I have 2020 set aside to implement siuba, a python port of the popular R library dplyr (siuba runs on top of pandas, but also can generate SQL). A huge source of inspiration has been screencasts by Dave Robinson using R to analyze data he's never seen before lightning fast.
Has anyone seen similar screencasts with pandas? I suspect it's not possible (given some constraints on its interface), but would love to be wrong here, because I'd like to keep all my work in python :o.
Expert R screencasts: https://youtu.be/NY0-IFet5AM
Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
#7We need a better decomposition of scalability. Do you mean scalability in data or scalability in compute or scalability of both?
Definitions:
Scalability in Data (SD): doing fast computation on a very large number of rows
Scalability in compute (SC): doing slow computations on a large number of rows
For SD, I have found that a 16-32 core machine is more than enough for tens of billions of rows as long as your disk access is relatively fast (SSD vs. HDD). If you vectorize your compute operations you can typically get to within 10x the assembly compute time. This allows you to tap into in a 32 core machine for 10s of effective giga flops. These machines are rated at 100s of giga flops. For example I had to compute a metric on a 100 million row table (dataframe) which effectively required on the order of 10-20 tflops of compute. Single-core pandas was showing us 2 months of compute time. Using vectorization and using mp.Pool I was able to reduce to a few hours. The big win here was vectorization and not mp.Pool.
For Compute scalability - e.g. running multiple machine learning models which cannot be effectively limited to a single machine, nothing beats Dask. Dask is extremely mature, has seen a large number of real world cases and people have used it for hundreds of hours of uptime.
Vectorization is a oft unlooked realm of speedup which can easily give you 10-100x speedups in Pandas. Understanding vectorization and what it can and cannot do is a highly productive exercise.
Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
#8Articles like these are interesting, but what surprises me is that they rarely set up a holistic use case, so most debates imagine how long it would take an expert user to use each tool. But time constraints (eg spent coding) separates expert from novice performance in many domains. FWIW I have 2020 set aside to implement siuba, a python port of the popular R library dplyr (siuba runs on top of pandas, but also can g…
David Robinson is great and you won't easily find material of the quality he produces elsewhere.
Take a look at Jake Vanderplas and Joel Grus's stuff for a general 'people doing cool things with Python' theme, but not quite comparable.
Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
#9Oh, I wrote this :) I submitted it last week but it didn't get much attention then. Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them. If you know more than I do and I made any mistakes, let me know and I'll get them corrected.
Do you also have experience working with SQL databases? If so, how do they compare to Pandas in terms of performance? (with or without these extensions)
It Depends (tm). I think SQL is one of the most underrated and underused languages and can often significantly out-perform Python for basic operations such as filtering and pivoting data.
That said, it's hard to keep SQL readable when doing more complicated data analysis, and you'll probably want the flexibility of Python the moment you start to do anything more custom.
Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids
#10Oh, I wrote this :) I submitted it last week but it didn't get much attention then. Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them. If you know more than I do and I made any mistakes, let me know and I'll get them corrected.
With Ray not having released a 1.0.0 version yet, does that give you any pause about adopting it for a professional project? In the article, you've given it an A for maturity, but the criteria didn't include versioning.
I've worked professionally with data scientists, and we've used both Dask and Ray with some success. Scaling pandas will be an issue for a long time to come with a lot of data science code being written in Python with Pandas.