Announcing Spark 1.3
databricks.com
Announcing Spark 1.3
1–10 of 24 posts
Re: Announcing Spark 1.3
#2Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.
Re: Announcing Spark 1.3
#3Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.
We are aware of at least 500 production use cases. Howeve,r since it is is open source software, there are a lot more that we don't know about.
You can find some public ones here:
http://spark-summit.org/east/2015/agenda
https://cwiki.apache.org/confluence/display/SPARK/Powered+By...
Re: Announcing Spark 1.3
#4Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.
Spark has a much nicer API than Hadoop and theoretically can give you significant speedups on iterative in-memory work loads. On the other hand, I've had a terrible experience with the stability and debuggability of previous versions of Spark. They do seem to be rapidly improving so it's hard to recommend anything other trying it out and seeing if Spark works for your needs.
(Also, you probably know this already, but many people don't really have data big enough to necessitate a distributed framework. If your datasets are counted in gigabytes, you can do everything more simply on one machine and/or with a traditional database)
Re: Announcing Spark 1.3
#5I guess the DataFrame API needs to be spelled out for me. Does this mean RDDs will be deprecated in the future if the new DataFrame is faster? As someone who's gateway drug into programming was R it's been fun to watch data frames grow across programming languages. I'm a huge fan of Python's Pandas library and very interested in Spark.
Re: Announcing Spark 1.3
#6I guess the DataFrame API needs to be spelled out for me. Does this mean RDDs will be deprecated in the future if the new DataFrame is faster? As someone who's gateway drug into programming was R it's been fun to watch data frames grow across programming languages. I'm a huge fan of Python's Pandas library and very interested in Spark.
The DataFrame is an evolution of the RDD model, where Spark knows explicit schema information. The core Spark RDD API is very generic and assumes nothing about the structure of the user's data. This is powerful, but ultimately the generic nature imposes limits on how much we can optimize.
DataFrames impose just a bit more structure: we assume that you have a tabular schema, named fields with types, etc. Given this assumption, Spark can optimize a lot of internal execution details, and also provide slicker API's to users. It turns out that a huge fraction of Spark workloads fall into this model, especially since we support complex types and nested structures.
Is the core RDD API going anywhere? Nope - not any time soon. Sometimes it really is necessary to drop into that lower level API. But I do anticipate that within a year or two most Spark applications will let DataFrames do the heavy lifting.
In fact, DataFrames and RDDs are completely inter-operable, either can be converted to the other. This means that even if you don't want to use DataFrames you can benefit from all of the cool input/output capabilities they have, even just to create regular old RDDs.
Re: Announcing Spark 1.3
#7I guess the DataFrame API needs to be spelled out for me. Does this mean RDDs will be deprecated in the future if the new DataFrame is faster? As someone who's gateway drug into programming was R it's been fun to watch data frames grow across programming languages. I'm a huge fan of Python's Pandas library and very interested in Spark.
The DataFrame is an evolution of the RDD model, where Spark knows explicit schema information. The core Spark RDD API is very generic and assumes nothing about the structure of the user's data. This is powerful, but ultimately the generic nature imposes limits on how much we can optimize. DataFrames impose just a bit more structure: we assume that you have a tabular schema, named fields with types, etc. Given this as…
> It turns out that a huge fraction of Spark workloads fall into this model, especially since we support complex types and nested structures.
The first step of all my Spark tasks is "turn this RDD[String] into an RDD of parsed JSON", or turning CSV into case classes.
What JSON parser will dataframes be using? I presume Jackson?
Re: Announcing Spark 1.3
#8Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.
Yep, we're using Spark 1.1 currently for aggregating log data on a one big bang per day basis, and I'm currently experimenting with Spark streaming also. I'd say go with Spark if it's green fields dev, the API is far nicer, and it's far less fiddly to work with.
Spark uses a lot of Hadoop under the covers, so you still benefit from that ecosystem.
What I really like about it is that it can be easily unit and integration tested.
Re: Announcing Spark 1.3
#9Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.
I believe one of the key advantages of Spark over Hadoop is being able to run the full stack on a small environment (single machine) and do all the coding there without the need of a cluster just for development.
Re: Announcing Spark 1.3
#10Out of curiosity, is anyone using Spark in production? We're evaluating whether we should invest in Hadoop or Spark. They're certainly not mutually exclusive, but I would rather invest fully in Spark than have infrastructure split between Spark and Hadoop.
Yep, we're using Spark 1.1 currently for aggregating log data on a one big bang per day basis, and I'm currently experimenting with Spark streaming also. I'd say go with Spark if it's green fields dev, the API is far nicer, and it's far less fiddly to work with. Spark uses a lot of Hadoop under the covers, so you still benefit from that ecosystem. What I really like about it is that it can be easily unit and integrat…
Interesting! Could you say more about the unit and integration testing? E.g., what sort of things you find it useful to test, and particular toolkits or approaches you like?