Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Spark: The Next Big Data Thing?

blog.mikiobraun.de

Post reply on HN
Print view View original

11–20 of 24 posts

Re: Apache Spark: The Next Big Data Thing?

#11

Post by etrain »

gizzlon wrote: post #10
tszming wrote: post #9

Earlier quoted context omitted.

The best way to try out Spark & related tools is to follow the 2 days exercises from the AMP Camp 3, they have a script to launch the whole cluster (Berkeley Data Analytics Stack) on ec2 very easily so you can get your feet wet without too much effort. [1] http://ampcamp.berkeley.edu/big-data-mini-course/ [2] http://ampcamp.berkeley.edu/big-data-mini-course/launching-a...

two days? :O

I helped write the AMPCamp exercises (and parts of Spark) - the exercises were done in the context of a two-day long seminar held at Berkeley where the days were mostly filled with talks/training about the stack. The actual exercises were designed to take a couple of hours each day, with plenty of time for questions/etc. You can probably get through them in an evening, and I've seen people do it in less!

Re: Apache Spark: The Next Big Data Thing?

#13

Post by TallGuyShort »

gclaramunt wrote: post #8
alanctgardner2 wrote: post #7

Earlier quoted context omitted.

In the next year I can imagine most people deploying Spark will be doing it on Hadoop, since Cloudera 5 will support Spark. It's a natural fit, most people don't hate HDFS but their use case doesn't naturally fit the MR programming model.

yeah, I mostly complain about trying to fit everything to the Map/Reduce model

The "Hadoop ecosystem" is getting less and less about MapReduce, and more about other execution frameworks that can share HDFS with it.

Re: Apache Spark: The Next Big Data Thing?

#14

Post by jey »

benjiweber wrote: post #3

Spark does look very promising. It is great being able to experiment with jobs on distributed collections from a Scala repl. Consequently it is very quick to get started. My first impressions are that there seems to be a fair amount of abstraction leakage. Some things that compile and look valid fail at runtime - e.g. referring to other RDDs from within a filter predicate or map function. Other More complex jobs caus…

    More complex jobs cause the nodes to fail and it gets
    into infinite loops of restarting the nodes, replaying 
    the job, and them dieing again.
This is probably a bug in your code. Debugging these cluster applications does take some getting used to. You'll want to look at the stderr output of the failing executor, and you'll probably see that it's dying due to some kind of exception. You can do this by visiting port 8080 of the master node over HTTP, i.e. http://mymaster:8080. Feel free to email the Spark users list if you have any questions: https://spark.incubator.apache.org/mailing-lists.html

It's true that you do have to understand the programming model and some details of how it's implemented to use Spark effectively. However, any abstraction that was "pure" and perfectly non-leaky would necessarily sacrifice some performance and transparency to achieve that goal. Spark aims to be both high-level and high-performance.

Full disclosure: I'm on the Spark team at the UC Berkeley AMPLab.

Re: Apache Spark: The Next Big Data Thing?

#15

Post by enigmo »

alanctgardner2 wrote: post #7
gclaramunt wrote: post #5

I love Spark, is about time to start the NoHadoop movement!

In the next year I can imagine most people deploying Spark will be doing it on Hadoop, since Cloudera 5 will support Spark. It's a natural fit, most people don't hate HDFS but their use case doesn't naturally fit the MR programming model.

I'd like to see other distributed filesystems catch on too. HDFS has a lot of room for improvement.

Re: Apache Spark: The Next Big Data Thing?

#16

Post by izendejas »

I believe the answer to that question is yes. I've raved about Spark many times on HN before. We're using it and absolutely love it.

Not sure why the author thinks there's no built-in support for iterations. The support is there, native scala/java. But you might have to collect to the driver/master to check for convergence, for example. Their simple logistic regression example doesn't check for convergence, but hard codes the number of iterations. http://spark.incubator.apache.org/examples.html

Some quirks that I've learned. When doing groupbys or other join operations, it helps to specify the number of partitions -- actually this is true in general, but more so for join-based operations, otherwise you can run into memory/gc issues. The equivalent is of course, setting the number of reducers to something which ensure you won't run out of heap space.

Because spark serializes the closures which transform your data, if you don't cache (at least via disk, using persist), then when iterating over an RDD or re-using it, you'll just waste cycles.

Beyond that, as an experienced scala developer, I've found that Spark feels incredibly natural. And being able to run things on the repl cannot be appreciated enough.

Re: Apache Spark: The Next Big Data Thing?

#17

Post by alanctgardner2 »

TallGuyShort wrote: post #13
gclaramunt wrote: post #8

Earlier quoted context omitted.

yeah, I mostly complain about trying to fit everything to the Map/Reduce model

The "Hadoop ecosystem" is getting less and less about MapReduce, and more about other execution frameworks that can share HDFS with it.

You don't happen to work at Cloudera, do you? I noticed you have some submissions about Impala and Oracle being evil, which seems to be a pretty common view among the ex-Oracle DBAs there

Re: Apache Spark: The Next Big Data Thing?

#18

Post by TallGuyShort »

alanctgardner2 wrote: post #17
TallGuyShort wrote: post #13

Earlier quoted context omitted.

The "Hadoop ecosystem" is getting less and less about MapReduce, and more about other execution frameworks that can share HDFS with it.

You don't happen to work at Cloudera, do you? I noticed you have some submissions about Impala and Oracle being evil, which seems to be a pretty common view among the ex-Oracle DBAs there

I do happen to work at Cloudera (hence the Impala submissions), although I'm neither an ex-Oracle DBA nor a huge believer that they're evil. I really don't have a lot of first-hand experience with Oracle as a company - which as you'll see, is why my submission was actually a question about the community's perception and why that's a common view.

Re: Apache Spark: The Next Big Data Thing?

#19

Post by jandrewrogers »

Spark cleans up some abstractions found in the Hadoop ecosystem but I would hesitate to call it the "next big thing" because it doesn't really address any of the core weaknesses of Hadoop. In Big Data, there are three touchstone applications that the current generation of platforms largely do poorly: real-time, geospatial, graph. Spark does not really address any of these. It might be more accurate to say that Spark is the "next big thing" in the Hadoop universe but still retains many of the same limitations on expressiveness, thereby creating opportunities for a more general Next Big Thing in Big Data.

Specifically, "faster batch + in-memory" is basically a simple patch on the batch mode problem but does not really address continuous flow real-time models. At large scales it is quite difficult to get robustly efficient behavior out of a parallel system that is batching things on 5 second intervals if the data flow is anything but trivial.

For geospatial and graph analysis, Spark appears to retain the same limitation of Hadoop in that it cannot deal with data models and operations without an a priori optimal partitioning function. Static hash and range partitioning won't cut it, particularly if the streaming data sources are actually real-time. The ability to generate uniform partitioning of complex data models with inherently unpredictable data distributions is critical to parallelizing some important analysis types but there is no obvious support for such mechanisms in Spark.

One could look at Spark as Hadoop done right, more or less.

Re: Apache Spark: The Next Big Data Thing?

#20

Post by monstrado »

enigmo wrote: post #15
alanctgardner2 wrote: post #7

Earlier quoted context omitted.

In the next year I can imagine most people deploying Spark will be doing it on Hadoop, since Cloudera 5 will support Spark. It's a natural fit, most people don't hate HDFS but their use case doesn't naturally fit the MR programming model.

I'd like to see other distributed filesystems catch on too. HDFS has a lot of room for improvement.

Which areas would you say HDFS needs improvement the most? Just so you know, HDFS is still very actively developed, and keeps introducing features / improving functionality (e.g Native NFS, In-Memory Caching, Short Circuit Reads, High Availability, Namespace Federation, etc) on a pretty regular basis.

Feel free to suggest new features, or contribute to the project yourself.

https://issues.apache.org/jira/browse/HDFS

Post reply on HN