21–30 of 47 posts
I'm one of the authors of the blog post as well as this new API. Feel free to ask me anything.
Are DataFrames RDDs with a new DSL?
Earlier quoted context omitted.
Are DataFrames RDDs with a new DSL?
In a way yes. It is a little bit more than that because DataFrames internally are actually "logical plans". Before execution, they are optimized by an optimizer called Catalyst and turn into physical plans.
I guess this means DataFrames should be used all the time in the future, or will there still be a reason to use plain RDDs in the future?
You guys are doing great work !
Earlier quoted context omitted.
Are DataFrames RDDs with a new DSL?
In a way yes. It is a little bit more than that because DataFrames internally are actually "logical plans". Before execution, they are optimized by an optimizer called Catalyst and turn into physical plans.
library(data.table)
x = data.table(a=sample(10,10e6,replace=TRUE),num=sample(100,10e6,replace=TRUE)) t1=proc.time(); x[,sum(num),by=a]; print(proc.time()-t1)
user system elapsed
0.209 0.032 0.245Earlier quoted context omitted.
In a way yes. It is a little bit more than that because DataFrames internally are actually "logical plans". Before execution, they are optimized by an optimizer called Catalyst and turn into physical plans.
Normal RDDs won't benefit from this optimisation, only DataFrames? Is that because using this new DSL allows Spark to more precisely plan what needs to happen for DataFrames? I guess this means DataFrames should be used all the time in the future, or will there still be a reason to use plain RDDs in the future? You guys are doing great work !
I'm one of the authors of the blog post as well as this new API. Feel free to ask me anything.
For some context - In our case, loading a reasonable set of data from HDFS can take upto 10-30 mins so keeping a cached copy of the most recent data with certain columns projected is important.
A replica of this benchmark on my laptop running R has this running in about 1/4 second. Seems like a pretty trivial benchmark? library(data.table) x = data.table(a=sample(10,10e6,replace=TRUE),num=sample(100,10e6,replace=TRUE)) t1=proc.time(); x[,sum(num),by=a]; print(proc.time()-t1) user system elapsed 0.209 0.032 0.245
I don't expect at small scale to beat R yet. There are a few low-hanging fruits for single node performance. For example, even for single node data, we incur a "shuffle" to do data exchange in aggregations. This is done to ensure both single node program and distributed program go through the same code path, to catch bugs. If we want to optimize more for single node performance, we can get the optimizer to remove the shuffle operation in the middle, and just run the aggregations. Then this toy example will probably be done in the 100ms range.
Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…
I don't think I could bring myself to ever write another Hadoop job.
Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…
It really is Hadoop 2.0.
Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…
Spark is less mature than Hadoop, so you will run into issues like this. In my experience, advocating for the bug to get fixed often results in it getting fixed... on a several month timeline. This happened with Avro support in Python. I advocated for the patch and someone supplied it in the next version of Spark. Lemme tell you though... as someone that has use Hadoop for 5+ years... not waiting 5-10 minutes every t…
And I love the fact you can press Tab and get autocompletion of methods.