21–30 of 58 posts
Thanks for sharing this. I'm the author of this blog post. Free free to ask me anything.
The TB sort benchmark is pretty useless to me - I am much more concerned with stability, a vibrant community (which means people, the software they write and institutions using Hadoop in production).
Last time I tinkered with Spark (this was over a year ago) it was so buggy, next to useless, but perhaps things have changed.
Still - the idea that there is some sort of a revolutionary new approach that is paradigm-shifting and is way better than anything before should be viewed with extreme skepticism.
The problem of distributed computing is not a simple one. I remember tinkering with the Linux kernel back in the mid nineties, and 20 years later it still has ways to go to improve.
Twenty years from now it might or might not be Hadoop that is the tool for this sort of thing, we don't know, but I will not take seriously anything or anyone who claims that the "next best thing" is here in 2014.
The strength of Hadoop isn't so much speed but that it's been around and there is a pretty impressive and fairly mature set of projects that comprises the Hadoop ecosystem, from Yarn to Hive, etc. There are still many issues to resolve, and this evolution will continue for decades to come. The TB sort benchmark is pretty useless to me - I am much more concerned with stability, a vibrant community (which means people,…
2. Yes, Spark was/is buggy.
3. For me Spark is really paradigm shift, next generation framework compared to M/R
Thanks for sharing this. I'm the author of this blog post. Free free to ask me anything.
The strength of Hadoop isn't so much speed but that it's been around and there is a pretty impressive and fairly mature set of projects that comprises the Hadoop ecosystem, from Yarn to Hive, etc. There are still many issues to resolve, and this evolution will continue for decades to come. The TB sort benchmark is pretty useless to me - I am much more concerned with stability, a vibrant community (which means people,…
1. Cloudera left M/R for Spark, Mahout left M/R for Spark. Spark community will be huge soon. 2. Yes, Spark was/is buggy. 3. For me Spark is really paradigm shift, next generation framework compared to M/R
If by M/R you mean Hadoop - Cloudera has done no such thing, their largest customer base is Hadoop.
As to "paradigm shift", we're so early in this that I don't think there even is a paradigm to shift.
Earlier quoted context omitted.
1. Cloudera left M/R for Spark, Mahout left M/R for Spark. Spark community will be huge soon. 2. Yes, Spark was/is buggy. 3. For me Spark is really paradigm shift, next generation framework compared to M/R
Hadoop != M/R, FWIW. M/R support is left in Yarn for backwards compatibility mostly. If by M/R you mean Hadoop - Cloudera has done no such thing, their largest customer base is Hadoop. As to "paradigm shift", we're so early in this that I don't think there even is a paradigm to shift.
The strength of Hadoop isn't so much speed but that it's been around and there is a pretty impressive and fairly mature set of projects that comprises the Hadoop ecosystem, from Yarn to Hive, etc. There are still many issues to resolve, and this evolution will continue for decades to come. The TB sort benchmark is pretty useless to me - I am much more concerned with stability, a vibrant community (which means people,…
1. Cloudera left M/R for Spark, Mahout left M/R for Spark. Spark community will be huge soon. 2. Yes, Spark was/is buggy. 3. For me Spark is really paradigm shift, next generation framework compared to M/R
There is a place for arguing how effective Map/Reduce is, but it's been known for years that M/R is not the only, nor best general purpose algorithm for solving all problems. More and more tools these days do not use M/R, Spark including, and Spark certainly is no the first tool to provide an alternative to M/R. AFAIK Google has abandoned M/R years ago.
I just don't understand this constant boasting about Spark, it seems very suspicious to me.
Thanks for sharing this. I'm the author of this blog post. Free free to ask me anything.
Your post mentions "single root IO virtualization" as a factor in maximizing network performance. I am wondering what the impact of this was in your sorting. Do you have data for runs where you didn't enable this?
Thanks for sharing this. I'm the author of this blog post. Free free to ask me anything.
Hi Reynold! Do you have numbers / intuition for how previous versions of spark would have run? I'm upgrading (soon) from spark 0.8 to spark 1.1 and am curious to see the performance gains (especially w.r.t. shuffles)
We haven't tested Spark 0.8 at this scale. In general Spark is advancing at a rapid rate that 1.1 is very very different from 0.8.
Earlier quoted context omitted.
1. Cloudera left M/R for Spark, Mahout left M/R for Spark. Spark community will be huge soon. 2. Yes, Spark was/is buggy. 3. For me Spark is really paradigm shift, next generation framework compared to M/R
Spark requires Hadoop to run, so this whole Spark vs Hadoop debate makes no sense whatsoever. There is a place for arguing how effective Map/Reduce is, but it's been known for years that M/R is not the only, nor best general purpose algorithm for solving all problems. More and more tools these days do not use M/R, Spark including, and Spark certainly is no the first tool to provide an alternative to M/R. AFAIK Google…
Spark does not need Hadoop - you can run it also with Mesos or in local mode..