Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

datarevenue.com

Post reply on HN
Print view View original

11–20 of 34 posts

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#11

Post by iwebdevfromhome »

sixhobbits wrote: post #4

Oh, I wrote this :) I submitted it last week but it didn't get much attention then. Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them. If you know more than I do and I made any mistakes, let me know and I'll get them corrected.

What are your thoughts on AWS Glue/Spark ? We’re starting to have problems with data frames that won’t fit into memory anymore on 32Gb clusters and upgrading to the next option, a 64Gb cluster, is an expensive thing. We plan to migrate to glue as a long term solution but I think we need to figure out a short term solution to the issue while the migration takes place.

Thanks for the article, before it I only knew of Dask as a real alternative.

P.D. I just remember that I wanted to try Pandarallel as well, so you have any insight on this library ? Thanks!

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#14

Post by ris »

sixhobbits wrote: post #9
erezsh wrote: post #5

Earlier quoted context omitted.

Do you also have experience working with SQL databases? If so, how do they compare to Pandas in terms of performance? (with or without these extensions)

It Depends (tm). I think SQL is one of the most underrated and underused languages and can often significantly out-perform Python for basic operations such as filtering and pivoting data. That said, it's hard to keep SQL readable when doing more complicated data analysis, and you'll probably want the flexibility of Python the moment you start to do anything more custom.

Therefore SQLAlchemy.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#15

Post by mumblemumble »

laactech wrote: post #10
sixhobbits wrote: post #4

Oh, I wrote this :) I submitted it last week but it didn't get much attention then. Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them. If you know more than I do and I made any mistakes, let me know and I'll get them corrected.

With Ray not having released a 1.0.0 version yet, does that give you any pause about adopting it for a professional project? In the article, you've given it an A for maturity, but the criteria didn't include versioning. I've worked professionally with data scientists, and we've used both Dask and Ray with some success. Scaling pandas will be an issue for a long time to come with a lot of data science code being writt…

That's such a tricky one, because the version number doesn't necessarily indicate the maturity.

For example, as of the beginning of this year, the latest version of Pandas was 0.25. ((t jumped to 1.0 in late January.) This despite it having been a core part of the standard professional toolset for years and years now.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#16

Post by jzwinck »

The subtitle is "How can you process more data quicker?"

NumPy. It scores an A in Maturity and Popularity, and either an A or a B in Ease of Adoption depending on which Pandas features you use (e.g. GroupBy).

When you're using NumPy as the main show instead of an implementation detail inside Pandas, it is easier to adopt Numba or Cython, and there are huge gains to be made there. Most Pandas workloads on small clusters of say 10 machines or fewer could be implemented on a single machine.

Even simple operations on smallish data sets are often much faster in NumPy than Pandas.

You don't have to leave Pandas behind, just try using NumPy and Numba for the hot parts of your code. Numba even lets you write Python code that works with the GIL released, which can lead to linear speedup in the number of cores with much less work than multiprocessing without the overhead of copying data to multiple processes.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#17

Post by haltingproblem »

jzwinck wrote: post #16

The subtitle is "How can you process more data quicker?" NumPy. It scores an A in Maturity and Popularity, and either an A or a B in Ease of Adoption depending on which Pandas features you use (e.g. GroupBy). When you're using NumPy as the main show instead of an implementation detail inside Pandas, it is easier to adopt Numba or Cython, and there are huge gains to be made there. Most Pandas workloads on small cluste…

This does not solve the issue of compute scalability - slow computations, which are fundamentally opaque, applied to large data frames . Given a series of data frames (or one large one that can be chunked) how do I apply a long running function to each chunk. For that you need scalability across cores and machines hence Dask.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#19

Post by vasili111 »

sixhobbits wrote: post #4

Oh, I wrote this :) I submitted it last week but it didn't get much attention then. Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them. If you know more than I do and I made any mistakes, let me know and I'll get them corrected.

What you think about Spark and PySpark?

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#20

Post by ramraj07 »

iwebdevfromhome wrote: post #11
sixhobbits wrote: post #4

Oh, I wrote this :) I submitted it last week but it didn't get much attention then. Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them. If you know more than I do and I made any mistakes, let me know and I'll get them corrected.

What are your thoughts on AWS Glue/Spark ? We’re starting to have problems with data frames that won’t fit into memory anymore on 32Gb clusters and upgrading to the next option, a 64Gb cluster, is an expensive thing. We plan to migrate to glue as a long term solution but I think we need to figure out a short term solution to the issue while the migration takes place. Thanks for the article, before it I only knew of D…

Be very, very sure that you understand how expensive Glue can get, especially with suboptimal code. I have seen bills 10x of same code running on emr spark clusters.
Post reply on HN