Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Spark 1.0.0

spark.apache.org

Post reply on HN
Print view View original

31–40 of 41 posts

Re: Apache Spark 1.0.0

#31

Post by iskander »

nemothekid wrote: post #11
agibsonccc wrote: post #7

Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though. Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..

Ooyala has a huge deployment that they use alongside their Cassandra cluster (something like ~100 nodes, and ~50TB of data IIRC)

I'd love to learn more about how they're using Spark, are there are any blog posts or tech talks floating around?

Re: Apache Spark 1.0.0

#32

Post by nemothekid »

iskander wrote: post #31
nemothekid wrote: post #11

Earlier quoted context omitted.

Ooyala has a huge deployment that they use alongside their Cassandra cluster (something like ~100 nodes, and ~50TB of data IIRC)

I'd love to learn more about how they're using Spark, are there are any blog posts or tech talks floating around?

Here's a talk at Hakka Labs done by a Ooyala Engineer (@evanfchan), which is how I knew they used Spark: https://www.youtube.com/watch?v=PjZp7K5z7ew - and the accompanying slides: http://www.slideshare.net/planetcassandra/south-bay-cassandr...

They use Spark on top of Cassandra, as well as they are users of Spark's version of Hive - Shark.

Re: Apache Spark 1.0.0

#34

Post by frak_your_couch »

espeed wrote: post #30
agibsonccc wrote: post #7

Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though. Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..

Cloudera now includes Spark... http://vision.cloudera.com/mapreduce-spark/ http://blog.cloudera.com/blog/category/spark/

So does Hortonworks as of 2.1

http://hortonworks.com/blog/announcing-hdp-2-1-tech-preview-...

Re: Apache Spark 1.0.0

#35

Post by espeed »

agibsonccc wrote: post #7

Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though. Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..

eBay posted this two days ago...

Using Spark to Ignite Data Analytics ( http://www.ebaytechblog.com/2014/05/28/using-spark-to-ignite...)

Re: Apache Spark 1.0.0

#36

Post by terranstyler »

gphil wrote: post #6
kovrik wrote: post #3

Any active Clojure bindings? clj-spark seems to be abandoned (last commit was a year ago)...

I'm curious about this too--clj-spark didn't work for me so I'm currently prototyping a job using the Java bindings in Clojure. If I end up wrapping the Java bindings in a useful way I would consider putting together some kind of release if there's community interest in that.

Me too.

However, shouldn't there be a much more flexible approach where you just send your functions to an execution server (just like an agent)? You might want to define some keywords to refer to previously used functions or data.

Then again, such an agent would be pretty much a REPL, so you might just want to ssh to a REPL that does load balancing and has sub-repls (on other machines) that fail over.

Thoughts on that?

Re: Apache Spark 1.0.0

#37

Post by lmm »

eranation wrote: post #9

I wonder if anyone with experience with Spark can comment / rebut this post: http://blog.explainmydata.com/2014/05/spark-should-be-better...

Spark's abstractions are indeed really nice; a Spark job is much more readable than the same thing expressed in raw MapReduce, as that post acknowledges at the end.

I can't really comment on or rebut "my code runs slow and I don't know why", except to say that Spark performance has been great when I've used it. But yeah, if the abstraction should fail (and again all I can say is it hasn't for me) then I can imagine it's not much fun to debug performance and there's no distributed profiler (though I think you'd be in much the same boat with vanilla Hadoop).

Re: Apache Spark 1.0.0

#38

Post by iskander »

lmm wrote: post #37
eranation wrote: post #9

I wonder if anyone with experience with Spark can comment / rebut this post: http://blog.explainmydata.com/2014/05/spark-should-be-better...

Spark's abstractions are indeed really nice; a Spark job is much more readable than the same thing expressed in raw MapReduce, as that post acknowledges at the end. I can't really comment on or rebut "my code runs slow and I don't know why", except to say that Spark performance has been great when I've used it. But yeah, if the abstraction should fail (and again all I can say is it hasn't for me) then I can imagine i…

> that Spark performance has been great when I've used it.

Can you say more about your use case? What sort of data did you start with? What did you do with it? How large was the cluster you were running on?

Re: Apache Spark 1.0.0

#39

Post by iskander »

nemothekid wrote: post #32
iskander wrote: post #31

Earlier quoted context omitted.

I'd love to learn more about how they're using Spark, are there are any blog posts or tech talks floating around?

Here's a talk at Hakka Labs done by a Ooyala Engineer (@evanfchan), which is how I knew they used Spark: https://www.youtube.com/watch?v=PjZp7K5z7ew - and the accompanying slides: http://www.slideshare.net/planetcassandra/south-bay-cassandr... They use Spark on top of Cassandra, as well as they are users of Spark's version of Hive - Shark.

Thanks for posting this. I'm starting to get a feel for when Spark is usable-- you need an underlying indexed data store which lets you fetch small subsets of your data into RDDs (or, your data can be tiny to begin with). We've been trying to use Spark on input sizes which, while smaller than our cluster's available memory, are probably too big for Spark to handle (> 1TB).

Re: Apache Spark 1.0.0

#40

Post by lmm »

iskander wrote: post #38
lmm wrote: post #37

Earlier quoted context omitted.

Spark's abstractions are indeed really nice; a Spark job is much more readable than the same thing expressed in raw MapReduce, as that post acknowledges at the end. I can't really comment on or rebut "my code runs slow and I don't know why", except to say that Spark performance has been great when I've used it. But yeah, if the abstraction should fail (and again all I can say is it hasn't for me) then I can imagine i…

> that Spark performance has been great when I've used it. Can you say more about your use case? What sort of data did you start with? What did you do with it? How large was the cluster you were running on?

Not sure how much I should say. Advertising analytics. Fairly small cluster (<100). More for ad-hoc theory testing rather than anything regular.
Post reply on HN