Apache Spark: The Next Big Data Thing? • Hacker News Board

Re: Apache Spark: The Next Big Data Thing?

#21

Post by espeed » Tue, Jan 21, 2014, 12:29 AM UTC

jandrewrogers wrote: post #19

Spark cleans up some abstractions found in the Hadoop ecosystem but I would hesitate to call it the "next big thing" because it doesn't really address any of the core weaknesses of Hadoop. In Big Data, there are three touchstone applications that the current generation of platforms largely do poorly: real-time, geospatial, graph. Spark does not really address any of these. It might be more accurate to say that Spark…

GraphX (http://amplab.github.io/graphx/) is essentially GraphLab2 (http://graphlab.org/) built on Spark. Joey Gonzalez, the creator of GraphLab, is also the GraphX lead.

Here are the papers for GraphLab and GraphX...

GraphLab: http://graphlab.org/home/publications/

GraphX: A Resilient Distributed Graph System on Spark (https://amplab.cs.berkeley.edu/publication/graphx-grades/)

See also "Introduction to GraphX - Presented by Joseph Gonzalez, Reynold Xin - UC Berkeley AmpLab 2013" (http://www.youtube.com/watch?v=mKEn9C5bRck)

Re: Apache Spark: The Next Big Data Thing?

#22

Post by bcbrown » Tue, Jan 21, 2014, 2:32 AM UTC

TallGuyShort wrote: post #18

alanctgardner2 wrote: post #17

Earlier quoted context omitted.

You don't happen to work at Cloudera, do you? I noticed you have some submissions about Impala and Oracle being evil, which seems to be a pretty common view among the ex-Oracle DBAs there

I do happen to work at Cloudera (hence the Impala submissions), although I'm neither an ex-Oracle DBA nor a huge believer that they're evil. I really don't have a lot of first-hand experience with Oracle as a company - which as you'll see, is why my submission was actually a question about the community's perception and why that's a common view.

Hey, I've recently begun the interview process at Cloudera, would you mind sending me an email? My address is in my profile, and I'd love to ask you a couple questions.

Re: Apache Spark: The Next Big Data Thing?

#23

Post by dundun » Tue, Jan 21, 2014, 6:31 AM UTC

alanctgardner2 wrote: post #7

gclaramunt wrote: post #5

I love Spark, is about time to start the NoHadoop movement!

In the next year I can imagine most people deploying Spark will be doing it on Hadoop, since Cloudera 5 will support Spark. It's a natural fit, most people don't hate HDFS but their use case doesn't naturally fit the MR programming model.

We've been using it on HDP2 for about a month now. Everything works fairly well and it was super easy to setup because of YARN (and the work the Spark team put in).

Re: Apache Spark: The Next Big Data Thing?

#24

Post by enigmo » Tue, Jan 21, 2014, 6:47 AM UTC

monstrado wrote: post #20

enigmo wrote: post #15

Earlier quoted context omitted.

I'd like to see other distributed filesystems catch on too. HDFS has a lot of room for improvement.

Which areas would you say HDFS needs improvement the most? Just so you know, HDFS is still very actively developed, and keeps introducing features / improving functionality (e.g Native NFS, In-Memory Caching, Short Circuit Reads, High Availability, Namespace Federation, etc) on a pretty regular basis. Feel free to suggest new features, or contribute to the project yourself. https://issues.apache.org/jira/browse/HDFS

I know of the JIRA, thanks. I'm aware that it is being actively developed, but I don't necessarily believe that activity is progress. Keeping a few different players in the mix helps keep everyone focused on progress.