Apache Spark: The Next Big Data Thing?
blog.mikiobraun.de
Apache Spark: The Next Big Data Thing?
1–10 of 24 posts
Re: Apache Spark: The Next Big Data Thing?
#2MLlib [1] is definitely worth checking out as a Spark library for
common ML algorithms - it's crazy how little code is needed to
implement efficient, distributed inference algorithms.
Spark's primitives (especially the RDD abstraction) make it feel like a DSL for distributed ML algorithms - for example, check out the implementation of distributed alternating least squares matrix factorization at [2] - ~100 lines of Scala code once stripped of the Java interoperability boilerplate.
On the weekend, I took a crack at implementing some distributed ML algorithms using Alternating Direction of Multipliers (ADMM) in Spark, following [3]. In a few hours and ~400 lines of Scala code, it was possible to implement distributed versions of L^1 regularized logistic regression, ridge regression, and SVMs [4].
It's a very impressive framework, and very much empowering.
[1]: http://spark.incubator.apache.org/docs/latest/mllib-guide.ht...
[2]: https://github.com/apache/incubator-spark/blob/fdaabdc673875...
[3]: http://www.stanford.edu/~boyd/papers/pdf/admm_distr_stats.pd...
[4]: https://github.com/ajtulloch/admmlrspark/tree/master/src/mai...
Re: Apache Spark: The Next Big Data Thing?
#3Spark does look very promising. It is great being able to experiment with jobs on distributed collections from a Scala repl. Consequently it is very quick to get started.
My first impressions are that there seems to be a fair amount of abstraction leakage. Some things that compile and look valid fail at runtime - e.g. referring to other RDDs from within a filter predicate or map function. Other More complex jobs cause the nodes to fail and it gets into infinite loops of restarting the nodes, replaying the job, and them dieing again.
I hope once I get a better understanding of what is going on underneath I will understand what is going on here.
Re: Apache Spark: The Next Big Data Thing?
#4Spark does look very promising. It is great being able to experiment with jobs on distributed collections from a Scala repl. Consequently it is very quick to get started. My first impressions are that there seems to be a fair amount of abstraction leakage. Some things that compile and look valid fail at runtime - e.g. referring to other RDDs from within a filter predicate or map function. Other More complex jobs caus…
Sounds a bit buggy, but the minimal API is very appealing.
Re: Apache Spark: The Next Big Data Thing?
#5I love Spark, is about time to start the NoHadoop movement!
Re: Apache Spark: The Next Big Data Thing?
#6I would like to play with Spark, have anyone tried any of the Docker images? There's too many =/
Re: Apache Spark: The Next Big Data Thing?
#7I love Spark, is about time to start the NoHadoop movement!
In the next year I can imagine most people deploying Spark will be doing it on Hadoop, since Cloudera 5 will support Spark. It's a natural fit, most people don't hate HDFS but their use case doesn't naturally fit the MR programming model.
Re: Apache Spark: The Next Big Data Thing?
#8I love Spark, is about time to start the NoHadoop movement!
In the next year I can imagine most people deploying Spark will be doing it on Hadoop, since Cloudera 5 will support Spark. It's a natural fit, most people don't hate HDFS but their use case doesn't naturally fit the MR programming model.
yeah, I mostly complain about trying to fit everything to the Map/Reduce model
Re: Apache Spark: The Next Big Data Thing?
#9I would like to play with Spark, have anyone tried any of the Docker images? There's too many =/
The best way to try out Spark & related tools is to follow the 2 days exercises from the AMP Camp 3, they have a script to launch the whole cluster (Berkeley Data Analytics Stack) on ec2 very easily so you can get your feet wet without too much effort.
[1] http://ampcamp.berkeley.edu/big-data-mini-course/
[2] http://ampcamp.berkeley.edu/big-data-mini-course/launching-a...
Re: Apache Spark: The Next Big Data Thing?
#10I would like to play with Spark, have anyone tried any of the Docker images? There's too many =/
The best way to try out Spark & related tools is to follow the 2 days exercises from the AMP Camp 3, they have a script to launch the whole cluster (Berkeley Data Analytics Stack) on ec2 very easily so you can get your feet wet without too much effort. [1] http://ampcamp.berkeley.edu/big-data-mini-course/ [2] http://ampcamp.berkeley.edu/big-data-mini-course/launching-a...
two days? :O