Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introducing DataFrames in Spark for Large Scale Data Science

databricks.com

Post reply on HN
Print view View original

11–20 of 47 posts

Re: Introducing DataFrames in Spark for Large Scale Data Science

#11

Post by rjurney »

pixelmonkey wrote: post #5
super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

Our experience was as a Python shop who was backed into a corner to use Apache Pig for our Hadoop batch jobs. We decided to rewrite some of those jobs from Pig to PySpark, and though there was a little bit of a learning curve and some sharp edges, the development experience is so much better than Pig that my team is generally happy with the switch.

PySpark is really compelling for Pig/Python shops. If it weren't for Pig on Spark, I'd fear for Pig's future.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#13

Post by rxin »

zodvik wrote: post #7

Will the DataFrame API work with Spark Streaming?

There are some ways you can integrate the two. E.g. Streaming allows you to apply arbitrary RDD transformations, and thus you can pass a physical plan generated by DataFrame into streaming.

We will work on better integration in the future too.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#15

Post by agibsonccc »

super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

I'm building distributed ml on top of spark and found it to be good overall. I've had to work out issues with partitioning and mini batching, but I've had a good time so far. The data frame initiative will certainly help things. The JVM ecosystem needs a scientific environment like python (pandas,scipy,..). The potential is there with scala as we're seeing here today.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#17

Post by super_sloth »

pwendell wrote: post #4
super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

Hey - sorry you had a bad experience. That bug was filed as a "minor" issue with only one user ever reporting it, so it didn't end up high up in our triage. We didn't merge the pull request because it was not correct, however, we can just add our own fix for it if it's affecting users. In the future, if you chime in on a reported JIRA, it will escalate it in our process.

Sorry, didn't mean to come over completely negative.

I appreciate the work that's gone in to Spark and it's clearly well designed. Developing with Spark after coming from a Hadoop background was a very refreshing experience.

Re: Introducing DataFrames in Spark for Large Scale Data Science

#19

Post by zgm »

super_sloth wrote: post #3

Has anyone had some good experiences with Spark? I put several weeks in to moving our machine learning pipeline over to Spark only to find I kept hitting a race condition in their scheduler. After doing a bit of searching, it seems this is actually a known issue https://issues.apache.org/jira/browse/SPARK-4454 and there's been a fix on their github for a while: https://github.com/apache/spark/pull/3345 and yet in tha…

We've recently adopted Spark SQL and our queries run 5 - 200x times faster than with Hive.

Our experience with Spark Streaming, on the other hand, has been mixed. Our streaming app runs stably most of the time (up to 4 days in some cases), but we still see the occasional failure, sometimes with no exception or stack trace indicating what failed.

Our goal is to have a 24/7 streaming service, and Spark has gotten us close to that. There are just a couple of unexplained errors standing in our way.

Post reply on HN