11–20 of 41 posts
Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though. Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..
I wonder if anyone with experience with Spark can comment / rebut this post: http://blog.explainmydata.com/2014/05/spark-should-be-better...
I could only speculate as to what this users issues were. One difference between hadoop and spark is that it is more sensitive in that you sometimes need to tell it how many tasks to use. In practice it is no big deal at all.
Perhaps the user was running into this- the data for a task in spark runs all in memory, whereas hadoop will load and spill to disk within a task. So if you give a single hadoop reducer 1TB of data, it will complete after a very long time. In spark if you did this you would need to have 1TB of memory on the executor. I wouldn't give an executor/JVM anything over 10GB. So if you have lots of memory, just be sure to balance it with cores and executors.
I have seen spark use up all the inodes on systems before. A job with 1000 map and 1000 reduce tasks would create 1M spill files on disk. However that was on an earlier version of spark and I was using ext3. I think this has since been improved.
For me spark runs circles around hadoop.
Note that Spark 1.0.0 makes it possible to trivially submit spark jobs to an existing Hadoop cluster. It leverages HDFS to distribute archives (e.g. your app JAR) and store results / state / logs, and YARN to schedule itself and acquire compute resources. It's pretty amazing to see how you use Spark's API to write functional applications that are then distributed across multiple executors (e.g. when you use Spark's "…
Any active Clojure bindings? clj-spark seems to be abandoned (last commit was a year ago)...
I'm curious about this too--clj-spark didn't work for me so I'm currently prototyping a job using the Java bindings in Clojure. If I end up wrapping the Java bindings in a useful way I would consider putting together some kind of release if there's community interest in that.
Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though. Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..
May be of relevance: https://cwiki.apache.org/confluence/display/SPARK/Powered+By... I don't know what you would count as major deployment, but I've deployed a 30-node cluster on HW for running sub-second real-time adhoc queries. I've also run many smaller 10-20 node virtual clusters on open stack. It is a rock solid platform. Our hosted ops loves it because it just works. The amazing thing about spark is how insanel…
Great to hear success stories!
I wonder if anyone with experience with Spark can comment / rebut this post: http://blog.explainmydata.com/2014/05/spark-should-be-better...
subprotocol's point about specifying the number of tasks/data partitions to use is true - you need to manually set this in order to get good results even on a small dataset. However, other than that, spark will give you good results pretty much out of the box. More advanced features such as broadcast objects, cache operations, and custom serializers will further optimize your application, but are not critical when first starting out as the author seems to believe.
Spark is an interesting technology, from what I've heard it doesn't actually have traction in industry yet though. Anyone here actually using it in production? I know it's blazing fast etc, and I like it as a map reduce replacement. It has all the makings of a great distributed system, I'm still waiting to see a major deployment yet..
I wonder if anyone with experience with Spark can comment / rebut this post: http://blog.explainmydata.com/2014/05/spark-should-be-better...
I use spark a lot and my experience has been quite the opposite. The queries I run against spark are billions of events and results are sub-second. I could only speculate as to what this users issues were. One difference between hadoop and spark is that it is more sensitive in that you sometimes need to tell it how many tasks to use. In practice it is no big deal at all. Perhaps the user was running into this- the da…
This is interesting, I haven't gotten Spark to do anything at all in less than a second. How big is this dataset (what does each event consist of)? How is the data stored? How many machines / cores are running across? What sort of queries are you running?
>I could only speculate as to what this users issues were.
I'm the author of the above post and unfortunately I can also "only speculate" what my issues were. Maybe Spark doesn't like 100x growth in the size of an RDD using flatMap? Maybe large-scale joins don't work well? Who knows. The problem, however, definitely doesn't seem to be anything from the tuning guide(s).
I wonder if anyone with experience with Spark can comment / rebut this post: http://blog.explainmydata.com/2014/05/spark-should-be-better...
My experience using spark has also been nothing but positive. I recently built a similarity-based recommendation on Spark ( https://github.com/evancasey/sparkler ), and found it to be significantly faster than comparable implementations on Hadoop. subprotocol's point about specifying the number of tasks/data partitions to use is true - you need to manually set this in order to get good results even on a small dataset…
Earlier quoted context omitted.
I use spark a lot and my experience has been quite the opposite. The queries I run against spark are billions of events and results are sub-second. I could only speculate as to what this users issues were. One difference between hadoop and spark is that it is more sensitive in that you sometimes need to tell it how many tasks to use. In practice it is no big deal at all. Perhaps the user was running into this- the da…
>The queries I run against spark are billions of events and results are sub-second. This is interesting, I haven't gotten Spark to do anything at all in less than a second. How big is this dataset (what does each event consist of)? How is the data stored? How many machines / cores are running across? What sort of queries are you running? >I could only speculate as to what this users issues were. I'm the author of the…
I'd be interested to hear more about your use case and the problems you encountered. It's possible that you need to do some kind of .coalesce() operation to rebalance the partitions if you have unbalanced partition sizes.