Announcing Spark 1.3 • Hacker News Board

Re: Announcing Spark 1.3

#21

Post by djcater » Sat, Mar 14, 2015, 11:04 AM UTC

Thanks for the hard work. When can we get the long-awaited ORC file support, including predicate-pushdown?

Re: Announcing Spark 1.3

#22

Post by threeseed » Sat, Mar 14, 2015, 12:42 PM UTC

ampermad wrote: post #14

Our experience with Spark has been horrendous. Very unstable. Marginal improvements. Big hassle. Would strongly advise you to consider hadoop. We also used storm and found it to be much stable. Databricks makes a lot of noise though.

Have to disagree on all points there.

Spark on YARN has been very stable and zero hassle for us. Run one command and it is deployed around the cluster and up and running. For performance Spark absolutely destroys stock MapReduce/Tez especially if like us you have a cluster with lots of basically unused RAM.

And Spark SQL, Spark Shell are both fantastic additions to the Hadoop ecosystem.

Re: Announcing Spark 1.3

#23

Post by mdaniel » Sat, Mar 14, 2015, 9:56 PM UTC

threeseed wrote: post #19

thethimble wrote: post #2

Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.

We are using Spark in production at a very large enterprise. One thing that has really helped us is Spark Job Server from Ooyala. I love the way you can share SparkContexts and just track what is happening: http://engineering.ooyala.com/blog/open-sourcing-our-spark-j...

That PR referenced in their blog post is 404; the actual project code is here:

https://github.com/spark-jobserver/spark-jobserver#readme

Re: Announcing Spark 1.3

#24

Post by wpietri » Mon, Mar 16, 2015, 1:23 AM UTC

EdwardDiego wrote: post #12

wpietri wrote: post #10

Earlier quoted context omitted.

Interesting! Could you say more about the unit and integration testing? E.g., what sort of things you find it useful to test, and particular toolkits or approaches you like?

Well, we unit test each of the functions which are typically mapping, reducing or filtering. The only thing special we do here is explicitly test that any reduce function which is required to be associative is actually associative, due to a dumb bug I wrote once. With the integration test, we have structured our code so that the processing occurs in a function that takes an RDD and returns an RDD. We then start a Spa…

Thanks. That's very helpful.