Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Spark 1.0.0

spark.apache.org

Post reply on HN
Print view View original

1–10 of 41 posts

Re: Apache Spark 1.0.0

#4

Post by krallin »

Note that Spark 1.0.0 makes it possible to trivially submit spark jobs to an existing Hadoop cluster.

It leverages HDFS to distribute archives (e.g. your app JAR) and store results / state / logs, and YARN to schedule itself and acquire compute resources.

It's pretty amazing to see how you use Spark's API to write functional applications that are then distributed across multiple executors (e.g. when you use Spark's "filter" or a "map" operations, then the operation potentially gets distributed and distributed on totally different nodes).

Great tool — exciting to see it reach 1.0.0!

Re: Apache Spark 1.0.0

#5

Post by MoOmer »

For new entrants, here's an email I sent out to some colleagues of mine just getting into ML. I'm wrapping up a project that's using Mahout, and am getting into Spark & MLlib now. I've regurgitated this on reddit already.

I've been following Apache Spark [0], a new-ish Apache project created by UC Berkeley to replace Hadoop MapReduce [1], for about a month now; and, I finally got around to spending some time with it last night and earllllllly this morning.

Added into the Spark mix about a year ago was a strong Machine Learning library (MLlib) [2] similar to Mahout [3] that promises much better performance (comparable/better than Matlab [4]/Vowpal Wabbit [5])

MLlib is a lower level library, which offers a lot of control/power for developers. However, Berkeley's Amplab has also created a higher level abstraction layer for end users called MLI [6]. It's still being actively developed, and although updates are in the works, they haven't been made available to the public repository for a while [7]

Check out an introduction to the MLlib on youtube here: https://www.youtube.com/watch?v=IxDnF_X4M-8

Getting up to speed with Spark itself is really pain-free compared to some tools like Mahout etc. There's a quick-start guide for Scala [8], a getting started guide for Spark [9], and lots of other learning/community resources available for Spark [10] [11]

[0] http://spark.apache.org/

[1] http://hadoop.apache.org/

[2] http://spark.apache.org/mllib/

[3] https://mahout.apache.org/

[4] http://www.mathworks.com/products/matlab/

[5] https://github.com/JohnLangford/vowpal_wabbit/wiki

[6] http://www.mlbase.org/

[7] http://apache-spark-user-list.1001560.n3.nabble.com/Status-o...

[8] www.artima.com/scalazine/articles/steps.html

[9] http://spark.apache.org/docs/latest/quick-start.html

[10] http://ampcamp.berkeley.edu/4/exercises/

[11] https://spark.apache.org/community.html

Re: Apache Spark 1.0.0

#6

Post by gphil »

kovrik wrote: post #3

Any active Clojure bindings? clj-spark seems to be abandoned (last commit was a year ago)...

I'm curious about this too--clj-spark didn't work for me so I'm currently prototyping a job using the Java bindings in Clojure. If I end up wrapping the Java bindings in a useful way I would consider putting together some kind of release if there's community interest in that.

Re: Apache Spark 1.0.0

#7

Post by agibsonccc »

Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though.

Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..

Re: Apache Spark 1.0.0

#10

Post by subprotocol »

agibsonccc wrote: post #7

Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though. Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..

May be of relevance: https://cwiki.apache.org/confluence/display/SPARK/Powered+By...

I don't know what you would count as major deployment, but I've deployed a 30-node cluster on HW for running sub-second real-time adhoc queries. I've also run many smaller 10-20 node virtual clusters on open stack. It is a rock solid platform. Our hosted ops loves it because it just works.

The amazing thing about spark is how insanely expressive and hackable it is. The best way I can describe it is this:

* Hadoop: You spend all of your time telling it how to do what you want (it is the assembly language of bigdata)

* Spark: you spend your time telling it what you want, and it just does it

Post reply on HN