Pandas on Ray – Early Lessons from Parallelizing Pandas
rise.cs.berkeley.edu
Pandas on Ray – Early Lessons from Parallelizing Pandas
1–10 of 20 posts
Re: Pandas on Ray – Early Lessons from Parallelizing Pandas
#2Code Repo: https://github.com/modin-project/modin
Re: Pandas on Ray – Early Lessons from Parallelizing Pandas
#3Code Repo: https://github.com/modin-project/modin
For those of you confused like me, "Pandas on Ray has moved into the Modin project"
Re: Pandas on Ray – Early Lessons from Parallelizing Pandas
#4Does anyone here know if Ray is some sort of Yarn competitor? If not what problem space is it in?
Re: Pandas on Ray – Early Lessons from Parallelizing Pandas
#5Unclear what this is good for.
Re: Pandas on Ray – Early Lessons from Parallelizing Pandas
#6Unclear what this is good for.
It is a WIP distributed Pandas implementation. Allows you to spread massive dataframes, their data and computations, over multiple machines vs. Pandas which is only local to your machine. It's not quite there yet [1]
[1] http://modin.readthedocs.io/en/latest/pandas_on_ray.html#usi...
Re: Pandas on Ray – Early Lessons from Parallelizing Pandas
#7Does anyone here know if Ray is some sort of Yarn competitor? If not what problem space is it in?
It feels to me like more of a spark competitor, aiming for the people who want more flexibility than Spark and not needing as much cluster setup (a redis cluster is the hardest part). It might even be able to use Yarn to do some of the resource management.
Re: Pandas on Ray – Early Lessons from Parallelizing Pandas
#8Does anyone here know if Ray is some sort of Yarn competitor? If not what problem space is it in?
I don't know what Yarn is, but Ray is a distributed heterogeneous computing framework. It's meant to make it easy to take fairly arbitrary computational programs (with machine learning being an important test/use case) and run/debug them in parallel across lots of machines with high performance in a natural fashion and without drastic changes. [1] The advantage compared to (say) Hadoop is that it allows for heterogeneous programming models and isn't limited to or designed for (say) MapReduce; you can invoke functions in parallel in a dynamic fashion pretty easily.
[1] You can get an idea of that here: https://ray.readthedocs.io/en/latest/
Re: Pandas on Ray – Early Lessons from Parallelizing Pandas
#9Unclear what this is good for.
It is a WIP distributed Pandas implementation. Allows you to spread massive dataframes, their data and computations, over multiple machines vs. Pandas which is only local to your machine. It's not quite there yet [1] [1] http://modin.readthedocs.io/en/latest/pandas_on_ray.html#usi...
Pandas replacing PySpark? Sign me up.
Re: Pandas on Ray – Early Lessons from Parallelizing Pandas
#10This could be really helpful for implementations that were written with relatively smaller datasets in mind but now need to be scaled up. However, for someone starting from scratch, it is not clear what advantages do they plan to offer against Spark used with the Dataframe API.