Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing Spark 1.3

databricks.com

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Announcing Spark 1.3

#11

Post by EdwardDiego »

gandalfu wrote: post #9
thethimble wrote: post #2

Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.

I believe one of the key advantages of Spark over Hadoop is being able to run the full stack on a small environment (single machine) and do all the coding there without the need of a cluster just for development.

This is a big sell for me. There are some small catches to it - for example, for operations requiring an associative function (such as reduceByKey), the need for it to be associative may not arise until the data-set becomes suitably large to be split across multiple workers, so in my team our testers specifically check reduce function associativity has been demonstrated in a unit test.

Re: Announcing Spark 1.3

#12

Post by EdwardDiego »

wpietri wrote: post #10
EdwardDiego wrote: post #8

Earlier quoted context omitted.

Yep, we're using Spark 1.1 currently for aggregating log data on a one big bang per day basis, and I'm currently experimenting with Spark streaming also. I'd say go with Spark if it's green fields dev, the API is far nicer, and it's far less fiddly to work with. Spark uses a lot of Hadoop under the covers, so you still benefit from that ecosystem. What I really like about it is that it can be easily unit and integrat…

Interesting! Could you say more about the unit and integration testing? E.g., what sort of things you find it useful to test, and particular toolkits or approaches you like?

Well, we unit test each of the functions which are typically mapping, reducing or filtering.

The only thing special we do here is explicitly test that any reduce function which is required to be associative is actually associative, due to a dumb bug I wrote once.

With the integration test, we have structured our code so that the processing occurs in a function that takes an RDD and returns an RDD. We then start a SparkContext in local mode[1], create an RDD with test data and can easily test that our processing produces the correct results.

We're just using JUnit for this, as we're largely a Java shop, so while we're coding in Scala for the cleaner API, we haven't jumped into the Scala ecosystem fully.

We also run end to end tests on our cluster using a subset of production data stored on S3 (Spark workers have to read/write from a distributed file system, S3 is one that the Hadoop ecosystem supports), and just verify the output against expectations derived from crunching that same subset via traditional means.

Hope that helps! I found Spark very easy to get up and running with, you can do a lot of experimentation in your IDE, and when you want to try a cluster, it ships with some convenience scripts that make it very easy to start a cluster on AWS.

[1]: http://spark.apache.org/docs/1.2.0/programming-guide.html#in...

Re: Announcing Spark 1.3

#13

Post by virmundi »

gandalfu wrote: post #9
thethimble wrote: post #2

Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.

I believe one of the key advantages of Spark over Hadoop is being able to run the full stack on a small environment (single machine) and do all the coding there without the need of a cluster just for development.

This is why I like cascading [1]. It has a higher level API on Hadoop. It also works in local mode with little change. I've actually used out to do transformation work from local files (csv), join them into structured documents and dump them into ArangoDB. I liked it so much I wrote a third party library to work ArangoDB in Hadoop[2].

1 cascading.org/ 2 https://github.com/deusdat/guacaphant

Re: Announcing Spark 1.3

#14

Post by ampermad »

Our experience with Spark has been horrendous. Very unstable. Marginal improvements. Big hassle.

Would strongly advise you to consider hadoop. We also used storm and found it to be much stable.

Databricks makes a lot of noise though.

Re: Announcing Spark 1.3

#15

Post by kiyoto »

ampermad wrote: post #14

Our experience with Spark has been horrendous. Very unstable. Marginal improvements. Big hassle. Would strongly advise you to consider hadoop. We also used storm and found it to be much stable. Databricks makes a lot of noise though.

Can you share the specifics of your experience (the cluster size/use case/etc.)

I am partly asking this because you clearly feel strongly about this topic (your account was created an hour ago, most likely to comment on this).

Re: Announcing Spark 1.3

#16

Post by lambdafunc »

gandalfu wrote: post #9
thethimble wrote: post #2

Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.

I believe one of the key advantages of Spark over Hadoop is being able to run the full stack on a small environment (single machine) and do all the coding there without the need of a cluster just for development.

You can also run mapreduce in local mode on a single machine.

Re: Announcing Spark 1.3

#17

Post by eranation »

thethimble wrote: post #2

Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.

We plan to... so far so good. Not in production yet though. We focus more on GraphX though, which is even less mature than Spark. We feel pretty comfortable with Spark core...

Re: Announcing Spark 1.3

#18

Post by eranation »

This is a really great release, I'm excited to start playing with DataFrames! One question if anyone from Databricks reads it - what about GraphX? Will it also get the same level of attention that SQL, Mlib and Spark core got recently? e.g. is adding support for Gremlin (or any other graph query languages) on the roadmap? what about an R API for GraphX? Is that planned? p.s. when is GraphX planned to exit Alpha?

in any case, great product, nice usage of Akka and Scala, and a very intuitive API. I feel lucky to be working with it on a daily basis.

Re: Announcing Spark 1.3

#19

Post by threeseed »

thethimble wrote: post #2

Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.

We are using Spark in production at a very large enterprise. One thing that has really helped us is Spark Job Server from Ooyala. I love the way you can share SparkContexts and just track what is happening:

http://engineering.ooyala.com/blog/open-sourcing-our-spark-j...

Re: Announcing Spark 1.3

#20

Post by rxin »

eranation wrote: post #18

This is a really great release, I'm excited to start playing with DataFrames! One question if anyone from Databricks reads it - what about GraphX? Will it also get the same level of attention that SQL, Mlib and Spark core got recently? e.g. is adding support for Gremlin (or any other graph query languages) on the roadmap? what about an R API for GraphX? Is that planned? p.s. when is GraphX planned to exit Alpha? in a…

GraphX actually graduated from Alpha in Spark 1.2.

We have a few important improvements and changes to GraphX planned for 1.4, including Java API, and possibly a Python API.

Post reply on HN