Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing DataFrames in Spark for Large Scale Data Science

databricks.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Introducing DataFrames in Spark for Large Scale Data Science

#31

Post by nchammas »

super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

Btw, there is an new PR by Josh Rosen to fix SPARK-4454 here: https://github.com/apache/spark/pull/4660

Re: Introducing DataFrames in Spark for Large Scale Data Science

#32

Post by rkrzr »

super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

We've been using it in production for about half a year now and it's been great. It especially shines when you have iterative algorithms were you e.g. first need to group-by something then process that a bit, then split it out again and process that a bit more etc. This kind of task is just so much faster in Spark than in MapReduce, it doesn't even compare. Also their API is much nicer and so are the underlying ideas (RDDs in particular). I think it will mostly replace MR in the future. If you are starting a new project now I see few reasons to not use Spark if it fits the bill.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#33

Post by rkrzr »

rxin wrote: post #25
century19 wrote: post #22

Earlier quoted context omitted.

Normal RDDs won't benefit from this optimisation, only DataFrames? Is that because using this new DSL allows Spark to more precisely plan what needs to happen for DataFrames? I guess this means DataFrames should be used all the time in the future, or will there still be a reason to use plain RDDs in the future? You guys are doing great work !

Indeed, DataFrames give Spark more semantic information about the data transformations, and thus can be better optimized. We envision this to become the primary API users use. You can still fall back to the vanilla RDD API (afterall DataFrame can be viewed as RDD[Row]) for stuff that is not expressible with DataFrames.

Could you give an example of something that could not be expressed with DataFrames? Would e.g. tree-structured data be a bad fit for DataFrames, since it doesn't fit well with the tabular nature?

Re: Introducing DataFrames in Spark for Large Scale Data Science

#35

Post by pwendell »

super_sloth wrote: post #17
pwendell wrote: post #4

Earlier quoted context omitted.

Hey - sorry you had a bad experience. That bug was filed as a "minor" issue with only one user ever reporting it, so it didn't end up high up in our triage. We didn't merge the pull request because it was not correct, however, we can just add our own fix for it if it's affecting users. In the future, if you chime in on a reported JIRA, it will escalate it in our process.

Sorry, didn't mean to come over completely negative. I appreciate the work that's gone in to Spark and it's clearly well designed. Developing with Spark after coming from a Hadoop background was a very refreshing experience.

No worries. Hopefully you'll reconsider using it!

Re: Introducing DataFrames in Spark for Large Scale Data Science

#36

Post by pwendell »

djcater wrote: post #34
rxin wrote: post #14

I'm one of the authors of the blog post as well as this new API. Feel free to ask me anything.

Are there any timelines for when this (and Spark in general) will fully support ORC files (including predicate-pushdown)?

Very likely in Spark 1.4. Hortonworks has been helping out with this, we just need some internal refacotring to the API to make it work.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#37

Post by eranation »

super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

I do, mostly using GraphX, ran a pretty big EC2 cluster using Spark 1.2 (hundreds of cores) and had no significant issues.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#40

Post by elliptic »

Spark the platform seems awesome. I'm somewhat less convinced by mllib - I'm not sure there are as many use cases for distributed machine learning as people seem to think (and I would bet that a good deal of companies that use distributed ML don't really need it). I've seen a lot of tasks that could be handled by simpler, faster algos on large workstations (you can get 250 GB RAM from AWS for like $4.00/hr). I'd love to hear counterarguments, though!
Post reply on HN