Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

datarevenue.com

Post reply on HN
Print view View original

21–30 of 34 posts

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#21

Post by ramraj07 »

ris wrote: post #14
sixhobbits wrote: post #9

Earlier quoted context omitted.

It Depends (tm). I think SQL is one of the most underrated and underused languages and can often significantly out-perform Python for basic operations such as filtering and pivoting data. That said, it's hard to keep SQL readable when doing more complicated data analysis, and you'll probably want the flexibility of Python the moment you start to do anything more custom.

Therefore SQLAlchemy.

I work with sqlalchemy a _lot_. Love it. But it's not intuitive at all especially for the majority of folks with a DS background.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#22

Post by nautilus12 »

vasili111 wrote: post #19
sixhobbits wrote: post #4

Oh, I wrote this :) I submitted it last week but it didn't get much attention then. Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them. If you know more than I do and I made any mistakes, let me know and I'll get them corrected.

What you think about Spark and PySpark?

Im going to give you my slightly biased and annoyed answer. It seems like people that use python tend to look down on spark as "too complicated" being written in Scala. I come from Scala background and now feeling forced into using python for my data work due to the momentum it has now I am still amazed at how quickly some simple requests like using a different image or having to attach some jars can make python people be like "whooa that's complicated, how can anyone like spark." Personally I love spark (for all it's quirks),and I think that the spark dataframe is much more mature in many ways to pandas, and the sanity type driven programming brings to table, and im kind of sad that im probably going to have to use python the rest of my career because there are so many fires it causes and a real strong tendency to kick many things down the road. The community just generally strikes me as very impatient.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#23

Post by sandGorgon »

The tie breaker here really is kubernetes. Most likely your company's infrastructure is run on k8s. As a data scientist you do not get control over that.

Dask natively integrates with Kubernetes. That's why I see a lot of people moving away even from Apache Spark (which is generally used through its inbuilt scheduler YARN) and towards Dask.

Second reason is that the dask-ml project is building seamless compatibility for higher order ML algorithms (sklearn,etc) on top of Dask. Not just Numpy/Pandas

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#24

Post by marcell »

I working on a project called CloudPy that's a 1-line drop in for pandas to process 100 GB+ dataframes.

It works by proxying the dataframe to a remote server, which can have a lot more memory than you're local server. The project is in beta right now, but please reach out if you're interested in trying it out! You can read more at http://www.cloudpy.io/ or email hello@cloudpy.io

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#25

Post by jzwinck »

haltingproblem wrote: post #17
jzwinck wrote: post #16

The subtitle is "How can you process more data quicker?" NumPy. It scores an A in Maturity and Popularity, and either an A or a B in Ease of Adoption depending on which Pandas features you use (e.g. GroupBy). When you're using NumPy as the main show instead of an implementation detail inside Pandas, it is easier to adopt Numba or Cython, and there are huge gains to be made there. Most Pandas workloads on small cluste…

This does not solve the issue of compute scalability - slow computations, which are fundamentally opaque, applied to large data frames . Given a series of data frames (or one large one that can be chunked) how do I apply a long running function to each chunk. For that you need scalability across cores and machines hence Dask.

Why do you consider computations to be opaque? Do you not have the source code?

There is a ton of low hanging speed in many computations that people treat as black boxes. Often as the result of knowing something extra about the specific input data rather than relying on a generic implementation.

In some cases all you need is to write NumPy code instead of Pandas code for a 2-3x speedup. Then suddenly your small cluster program runs on one machine.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#27

Post by isoprophlex »

blackbear_ wrote: post #26

I'll just throw it in the discussion: pandas could just interface with and leave the heavy lifting to a RDBMS.

In my opinion, pandas is fundamentally broken and unsuitable for any production workload.

The heavy lifting should be left to a RDBMS like you say: something with a sensible, battle-hardened query planner. I've written and debugged too many lines of manual pd joins/merges; something declarative like SQL is much nicer because the query planner is almost always right.

Furthermore, as a user, I've always found the pandas API to be very confusing. I'm always having to interrupt my workflow to figure out boring details about the API (is it df.groupBy().rolling(center=True).median() or any other permutation?), whereas eg pyspark or sql are so much more ergonomic.

Finally, typing inside pd dataframes is a complete and utter nightmare. Int64 missing a null, or the idiocy around datetimes expressed as epoch nanoseconds...

Pandas is nice for noodling around in notebooks. But for me, it should never be used beyond that.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#28

Post by isoprophlex »

jzwinck wrote: post #25
haltingproblem wrote: post #17

Earlier quoted context omitted.

This does not solve the issue of compute scalability - slow computations, which are fundamentally opaque, applied to large data frames . Given a series of data frames (or one large one that can be chunked) how do I apply a long running function to each chunk. For that you need scalability across cores and machines hence Dask.

Why do you consider computations to be opaque? Do you not have the source code? There is a ton of low hanging speed in many computations that people treat as black boxes. Often as the result of knowing something extra about the specific input data rather than relying on a generic implementation. In some cases all you need is to write NumPy code instead of Pandas code for a 2-3x speedup. Then suddenly your small clust…

Besides the speedup from using native numpy, theres also the potential for 50-100x speedup if your code isn't vectorized to begin with, and anywhere from 1-1000x if theres a couple of joins in there that you can optimize.

But for the latter, see discussion on shifting the pd compute to a RDBMS elsewhere in these comments.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#29

Post by disgruntledphd2 »

iwebdevfromhome wrote: post #11
sixhobbits wrote: post #4

Oh, I wrote this :) I submitted it last week but it didn't get much attention then. Happy to answer questions as far as possible. I have used Pandas extensively but I don't have deep experience with all of these libraries so I learnt a lot while summarising them. If you know more than I do and I made any mistakes, let me know and I'll get them corrected.

What are your thoughts on AWS Glue/Spark ? We’re starting to have problems with data frames that won’t fit into memory anymore on 32Gb clusters and upgrading to the next option, a 64Gb cluster, is an expensive thing. We plan to migrate to glue as a long term solution but I think we need to figure out a short term solution to the issue while the migration takes place. Thanks for the article, before it I only knew of D…

Not the OP, but moving to sparse matrices is probably going to give you the most bang for your buck. I would strongly suspect that those huge dataframes could be encoded sparsely in a much more efficient format.

To be fair, that's one of the reasons that Spark ML stuff works quite well. Be warned though, estimating how long a Spark job will take/how much resources it will need is a dark, dark art.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#30

Post by disgruntledphd2 »

nautilus12 wrote: post #22
vasili111 wrote: post #19

Earlier quoted context omitted.

What you think about Spark and PySpark?

Im going to give you my slightly biased and annoyed answer. It seems like people that use python tend to look down on spark as "too complicated" being written in Scala. I come from Scala background and now feeling forced into using python for my data work due to the momentum it has now I am still amazed at how quickly some simple requests like using a different image or having to attach some jars can make python peop…

Spark is really, really good. It's a massive leap from the Python/R model of play around with a data.frame till I have a model, then wrap it up in a script for a lot of data scientists though, which causes problems.

Spark is ace as it has an SQL API available cross-language, which makes ETL much more effective, and ML models (though I've always been sort-of suspicious about their maturity).

tl;dr - demonstrate the speed of running regressions in Spark, and many (most) data scientists will invest the time in learning the tool.

Post reply on HN