Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids • Hacker News Board

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#31

Post by disgruntledphd2 » Mon, Jul 06, 2020, 8:32 AM UTC

isoprophlex wrote: post #27

blackbear_ wrote: post #26

I'll just throw it in the discussion: pandas could just interface with and leave the heavy lifting to a RDBMS.

In my opinion, pandas is fundamentally broken and unsuitable for any production workload. The heavy lifting should be left to a RDBMS like you say: something with a sensible, battle-hardened query planner. I've written and debugged too many lines of manual pd joins/merges; something declarative like SQL is much nicer because the query planner is almost always right. Furthermore, as a user, I've always found the panda…

Pandas combines the intuitive nature of base-R with the excellent missing data model of Python.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#32

Post by haltingproblem » Mon, Jul 06, 2020, 4:17 PM UTC

jzwinck wrote: post #25

haltingproblem wrote: post #17

Earlier quoted context omitted.

This does not solve the issue of compute scalability - slow computations, which are fundamentally opaque, applied to large data frames . Given a series of data frames (or one large one that can be chunked) how do I apply a long running function to each chunk. For that you need scalability across cores and machines hence Dask.

Why do you consider computations to be opaque? Do you not have the source code? There is a ton of low hanging speed in many computations that people treat as black boxes. Often as the result of knowing something extra about the specific input data rather than relying on a generic implementation. In some cases all you need is to write NumPy code instead of Pandas code for a 2-3x speedup. Then suddenly your small clust…

SK Learn is the most popular ML libs. Well written, source code available, etc. But I am not opening it up to optimize it and neither should anyone unless they are already a SK Learn contributor OR have a ton of time on their hands.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#33

Post by prepend » Mon, Jul 06, 2020, 7:44 PM UTC

blackbear_ wrote: post #26

I'll just throw it in the discussion: pandas could just interface with and leave the heavy lifting to a RDBMS.

I use pandas with multiple sources and try to keep as much as possible in the db.

But many sources are outside the db, or run on multiple, disconnected dbs.

Loading data into a db is impractical and I only do it if necessary.

Also, different people run and manage dbs so it’s frequently easier to run in pandas.

Re: Scaling Pandas: Comparing Dask, Ray, Modin, Vaex, and Rapids

#34

Post by alpineidyll3 » Tue, Jul 07, 2020, 4:19 AM UTC

isoprophlex wrote: post #27

blackbear_ wrote: post #26

I'll just throw it in the discussion: pandas could just interface with and leave the heavy lifting to a RDBMS.

In my opinion, pandas is fundamentally broken and unsuitable for any production workload. The heavy lifting should be left to a RDBMS like you say: something with a sensible, battle-hardened query planner. I've written and debugged too many lines of manual pd joins/merges; something declarative like SQL is much nicer because the query planner is almost always right. Furthermore, as a user, I've always found the panda…

Aye aye!