Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Spark 1.0.0

spark.apache.org

Post reply on HN
Print view View original

21–30 of 41 posts

Re: Apache Spark 1.0.0

#21

Post by subprotocol »

iskander wrote: post #18
subprotocol wrote: post #12

Earlier quoted context omitted.

I use spark a lot and my experience has been quite the opposite. The queries I run against spark are billions of events and results are sub-second. I could only speculate as to what this users issues were. One difference between hadoop and spark is that it is more sensitive in that you sometimes need to tell it how many tasks to use. In practice it is no big deal at all. Perhaps the user was running into this- the da…

>The queries I run against spark are billions of events and results are sub-second. This is interesting, I haven't gotten Spark to do anything at all in less than a second. How big is this dataset (what does each event consist of)? How is the data stored? How many machines / cores are running across? What sort of queries are you running? >I could only speculate as to what this users issues were. I'm the author of the…

> How big is this dataset (what does each event consist of)?

Standard clickstream data, maybe 50-ish parameters per event.

> What sort of queries are you running? > How is the data stored?

Depends on the use-case. For sub-second adhoc queries we go against bitmap indexes. Other queries we uses RDD.cache() after a group/cogroup and answer queries directly from that. For other queries we go hit ORC files. Spark is very memory sensitive compared to hadoop, so using a columnar store and only pulling out the data that you absolutely need goes a very long way. Minimizing cross-communication and shuffling is key to achieving sub-second. It's impossible to achieve that if you're waiting for TB of data to shuffle around =)

> How many machines / cores are running across?

Depends on the use case. Clusters are 10-30 machines, some we run virtual on open stack. We will grow our 30 node cluster in 6mo.

> Maybe Spark doesn't like 100x growth in the size of an RDD using flatMap

You may actually just need to proportionally scale the number of partitions for that particular task by the same amount. Also when possible use mapPartitions, it is very memory efficient compared to map/flatMap.

> Maybe large-scale joins don't work well

Keep in mind that what ever happens per task happens all in memory. For large joins I created a "bloom join" implementation (not currently open source =( ) that does this efficiently. It takes two passes at the data, but minimizes what is shuffled.

Re: Apache Spark 1.0.0

#22

Post by evancasey »

iskander wrote: post #19
evancasey wrote: post #16

Earlier quoted context omitted.

My experience using spark has also been nothing but positive. I recently built a similarity-based recommendation on Spark ( https://github.com/evancasey/sparkler ), and found it to be significantly faster than comparable implementations on Hadoop. subprotocol's point about specifying the number of tasks/data partitions to use is true - you need to manually set this in order to get good results even on a small dataset…

I'm really curious to find out in what situations Spark actually works for people. So far, no one in my lab seems to be having a terribly productive time using it. Maybe it's better for simple numerical computations? How large are the datasets you're working with?

I did most of my benchmarking with the 10M MovieLens Dataset http://grouplens.org/datasets/movielens/ consisting of 10 million movie ratings on 10,000 movies from 72,000 users. So not necessarily "big data", but big enough to warrant a distributed approach.

Spark is ideally suited for iterative, multi-stage jobs. In theory, anything that requires doing multiple operations an a working dataset (i.e. graph processing, recommender systems, gradient descent) will do well on Spark due to the in-memory data caching model. This post explains some of the applications Spark is well-suited for: http://www.quora.com/Apache-Spark/What-are-use-cases-for-spa...

Re: Apache Spark 1.0.0

#23

Post by iskander »

evancasey wrote: post #22
iskander wrote: post #19

Earlier quoted context omitted.

I'm really curious to find out in what situations Spark actually works for people. So far, no one in my lab seems to be having a terribly productive time using it. Maybe it's better for simple numerical computations? How large are the datasets you're working with?

I did most of my benchmarking with the 10M MovieLens Dataset http://grouplens.org/datasets/movielens/ consisting of 10 million movie ratings on 10,000 movies from 72,000 users. So not necessarily "big data", but big enough to warrant a distributed approach. Spark is ideally suited for iterative, multi-stage jobs. In theory, anything that requires doing multiple operations an a working dataset (i.e. graph processing,…

So the central piece of data is something like a 10 million element RDD of (UserId, (MovieId, Rating))? If so, it sounds like that data would fit into a single in-memory sparse array, how does Spark's performance compare with a local implementation?

By comparison, I'm trying (and failing) to work with RDDs of 100+ billion elements.

Re: Apache Spark 1.0.0

#24

Post by iskander »

subprotocol wrote: post #10
agibsonccc wrote: post #7

Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though. Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..

May be of relevance: https://cwiki.apache.org/confluence/display/SPARK/Powered+By... I don't know what you would count as major deployment, but I've deployed a 30-node cluster on HW for running sub-second real-time adhoc queries. I've also run many smaller 10-20 node virtual clusters on open stack. It is a rock solid platform. Our hosted ops loves it because it just works. The amazing thing about spark is how insanel…

> It is a rock solid platform. Our hosted ops loves it because it just works.

I'm really confused by how different our experiences have been. Above you said you wrote your own shuffle implementation, presumably that was prompted by poor performance at some point. And, when you encountered that poor performance, you presumably also saw what happens to Spark when it's overwhelmed: a sea of exceptions. In a short period of time I've encountered lots of the following:

- FileNotFound exceptions when shuffle files couldn't be created

- Too many open file handles (also related to shuffle files)

- Infinite procession of out-of-memory errors on a cluster with 12TB of memory.

- Executor disconnected

- Weird akka errors

- Mysterious serialization errors (Map.values isn't serializable, making a nested partitioner class for some reason didn't work)

These errors are sometimes recoverable and other times kill all the workers on the cluster. Did none of these things happen to your team?

Re: Apache Spark 1.0.0

#25

Post by [deleted] »

agibsonccc wrote: post #15
subprotocol wrote: post #10

Earlier quoted context omitted.

May be of relevance: https://cwiki.apache.org/confluence/display/SPARK/Powered+By... I don't know what you would count as major deployment, but I've deployed a 30-node cluster on HW for running sub-second real-time adhoc queries. I've also run many smaller 10-20 node virtual clusters on open stack. It is a rock solid platform. Our hosted ops loves it because it just works. The amazing thing about spark is how insanel…

This does help actually. And yes: it doesn't have to be a 1000 node cluster or anything crazy. I've just talked to a lot of people at bigger companies and they've all said it falls over yet. Great to hear success stories!

[deleted]

Re: Apache Spark 1.0.0

#26

Post by platz »

iskander wrote: post #23
evancasey wrote: post #22

Earlier quoted context omitted.

I did most of my benchmarking with the 10M MovieLens Dataset http://grouplens.org/datasets/movielens/ consisting of 10 million movie ratings on 10,000 movies from 72,000 users. So not necessarily "big data", but big enough to warrant a distributed approach. Spark is ideally suited for iterative, multi-stage jobs. In theory, anything that requires doing multiple operations an a working dataset (i.e. graph processing,…

So the central piece of data is something like a 10 million element RDD of (UserId, (MovieId, Rating))? If so, it sounds like that data would fit into a single in-memory sparse array, how does Spark's performance compare with a local implementation? By comparison, I'm trying (and failing) to work with RDDs of 100+ billion elements.

What is the difference between Spark and Storm? They both seem like "realtime compute engines"

*edit - from what I can see Spark is a replacement for hadoop (offline jobs), where Storm deals with online stream processing

Re: Apache Spark 1.0.0

#27

Post by iskander »

subprotocol wrote: post #21
iskander wrote: post #18

Earlier quoted context omitted.

>The queries I run against spark are billions of events and results are sub-second. This is interesting, I haven't gotten Spark to do anything at all in less than a second. How big is this dataset (what does each event consist of)? How is the data stored? How many machines / cores are running across? What sort of queries are you running? >I could only speculate as to what this users issues were. I'm the author of the…

> How big is this dataset (what does each event consist of)? Standard clickstream data, maybe 50-ish parameters per event. > What sort of queries are you running? > How is the data stored? Depends on the use-case. For sub-second adhoc queries we go against bitmap indexes. Other queries we uses RDD.cache() after a group/cogroup and answer queries directly from that. For other queries we go hit ORC files. Spark is very…

> For sub-second adhoc queries we go against bitmap indexes.

Did you implement the index yourself? How many events survive the initial filtering?

Re: Apache Spark 1.0.0

#28

Post by iskander »

jey wrote: post #20
iskander wrote: post #18

Earlier quoted context omitted.

>The queries I run against spark are billions of events and results are sub-second. This is interesting, I haven't gotten Spark to do anything at all in less than a second. How big is this dataset (what does each event consist of)? How is the data stored? How many machines / cores are running across? What sort of queries are you running? >I could only speculate as to what this users issues were. I'm the author of the…

> Maybe Spark doesn't like 100x growth in the size of an RDD using flatMap? I'd be interested to hear more about your use case and the problems you encountered. It's possible that you need to do some kind of .coalesce() operation to rebalance the partitions if you have unbalanced partition sizes.

Well, the RDD is initially partitioned using a RangePartitioner over a dense key space of Longs. Each element is then expanded ~100x (each object is significantly smaller than the original value). So the total memory footprint and skew of the expanded RDD shouldn't, theoretically, be a problem.

Re: Apache Spark 1.0.0

#29

Post by ironchef »

platz wrote: post #26
iskander wrote: post #23

Earlier quoted context omitted.

So the central piece of data is something like a 10 million element RDD of (UserId, (MovieId, Rating))? If so, it sounds like that data would fit into a single in-memory sparse array, how does Spark's performance compare with a local implementation? By comparison, I'm trying (and failing) to work with RDDs of 100+ billion elements.

What is the difference between Spark and Storm? They both seem like "realtime compute engines" *edit - from what I can see Spark is a replacement for hadoop (offline jobs), where Storm deals with online stream processing

Storm is generally more of a dataflow "per event" real/near time computation system (with each event flowing through N spouts and bolts) whereas Spark is more of an in-memory data processing system (with Spark streaming being the "equivalent" to the storm system).

Re: Apache Spark 1.0.0

#30

Post by espeed »

agibsonccc wrote: post #7

Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though. Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..

Cloudera now includes Spark...

http://vision.cloudera.com/mapreduce-spark/

http://blog.cloudera.com/blog/category/spark/

Post reply on HN