Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing DataFrames in Spark for Large Scale Data Science

databricks.com

Post reply on HN
Print view View original

1–10 of 47 posts

Introducing DataFrames in Spark for Large Scale Data Science

#1

Post by rxin »

I'm the author of this blog post. We are very excited about this API and think it will be the common interchange format for data in Spark. It has also some neat features (such as code generation, predicate push down, etc) that make it very useful for Big Data.

Feel free to ask me anything.

Introducing DataFrames in Spark for Large Scale Data Science
databricks.com

Re: Introducing DataFrames in Spark for Large Scale Data Science

#3

Post by super_sloth »

Has anyone had some good experiences with Spark?

I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler.

After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in that time two releases have swung by bringing a tonne of features.

I ended up having to drop Spark ultimately because I wasn't confident about putting it in to production (the random OOMs and NPEs during development weren't great either). Does anyone have any positive experiences?

Re: Introducing DataFrames in Spark for Large Scale Data Science

#4

Post by pwendell »

super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

Hey - sorry you had a bad experience. That bug was filed as a "minor" issue with only one user ever reporting it, so it didn't end up high up in our triage. We didn't merge the pull request because it was not correct, however, we can just add our own fix for it if it's affecting users. In the future, if you chime in on a reported JIRA, it will escalate it in our process.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#5

Post by pixelmonkey »

super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

Our experience was as a Python shop who was backed into a corner to use Apache Pig for our Hadoop batch jobs.

We decided to rewrite some of those jobs from Pig to PySpark, and though there was a little bit of a learning curve and some sharp edges, the development experience is so much better than Pig that my team is generally happy with the switch.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#6

Post by TallGuyShort »

super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

I've used Spark quite successfully for a few small jobs and I love it. I'm sure robustness will improve over time (haven't been bothered by any immaturity myself), but as a system that supports a variety of Big Data processing styles like batch, streaming, graph, ML, etc. so well and is clearly bridging the gap between distributed systems and more traditional analysis languages, I think it's very exciting.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#8

Post by rjurney »

super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

Spark is less mature than Hadoop, so you will run into issues like this. In my experience, advocating for the bug to get fixed often results in it getting fixed... on a several month timeline. This happened with Avro support in Python. I advocated for the patch and someone supplied it in the next version of Spark.

Lemme tell you though... as someone that has use Hadoop for 5+ years... not waiting 5-10 minutes every time you run new code is worth the trouble. Despite more problems owing to immaturity, or just Spark 'doing less for you' in terms of data validation than other tools like Pig/Hive, if you can get your stuff running on Spark... development is joyous. You just don't have to wait very long during development anymore.

I feel like 5 years of my life were delayed 10 minutes. That did terrible things to my coding that I'm just starting to get over. With Spark I am 10x as productive, and I am limited by my thinking, not the tools.

PySpark in particular is really great.

Post reply on HN