41–50 of 58 posts
I wrote a post on this: http://www.silota.com/site-search-blog/approximate-median-co...
Going on a tangent here: this benchmark highlights the difficulty of sorting in general. Sorts are necessary for computing percentiles (such as the median.) In practical applications, an approximate algorithm such as t-digest should suffice. You can return results in seconds as opposed to "chest thumping" benchmarks to prove a point. :) I wrote a post on this: http://www.silota.com/site-search-blog/approximate-median…
Earlier quoted context omitted.
1. Cloudera left M/R for Spark, Mahout left M/R for Spark. Spark community will be huge soon. 2. Yes, Spark was/is buggy. 3. For me Spark is really paradigm shift, next generation framework compared to M/R
Spark requires Hadoop to run, so this whole Spark vs Hadoop debate makes no sense whatsoever. There is a place for arguing how effective Map/Reduce is, but it's been known for years that M/R is not the only, nor best general purpose algorithm for solving all problems. More and more tools these days do not use M/R, Spark including, and Spark certainly is no the first tool to provide an alternative to M/R. AFAIK Google…
This is not correct. Spark uses the Hadoop Input/Output API, but you don't need any Hadoop component installed to run Spark, not even HDFS.
You can -- and many companies do -- run Spark on Mesos or on Spark's standalone cluster manager, and use S3 as their storage layer.
> this whole Spark vs Hadoop debate makes no sense whatsoever
If we talk about Hadoop as an ecosystem of tools, then yes, it doesn't make sense to frame Spark as a competitor. Spark is part of that ecosystem.
But if we talk about Hadoop as Hadoop 1 MapReduce or as Hadoop 2 Tez, both of which are execution engines, then it very much makes sense to pit Spark against them as an alternative execution engine.
Granted, Hadoop 1 MapReduce is pretty old compared to Spark, and Tez is still under heavy development, but these are alternatives and not complements to Spark.
(Note: In Hadoop 2, MapReduce is just a framework that uses Tez as its underlying execution engine.)
> I just don't understand this constant boasting about Spark, it seems very suspicious to me.
Suspicious how?
I think Spark's elegant API, unified data processing model, and performance -- all of which are documented very well in demos and benchmarks online -- merit the excitement that you see in the "Big Data" community.
It's interesting, but not earth-shattering. The "10x fewer nodes" means nothing; how powerful are the new nodes? What's the network? Do you use SSDs? etc. etc. They also tuned their code to this specific problem: " Exploiting Cache Locality: In the sort benchmark, each record is 100 bytes, where the sort key is the first 10 bytes. As we were profiling our sort program, we noticed the cache miss rate was high, because…
You may find this benchmark [1] interesting to read.
It needs some updating (a lot has changed since February 2014), but it compares Shark (which uses Spark as its execution engine) to Hive (using Hadoop 1 MapReduce as its execution engine) and a number of other systems.
The benchmark is run on EC2 and is detailed in such a way that it should be independently verifiable. Hive and Shark are run on identically sized clusters, though I don't know if the other details of the configuration were identical.
The strength of Hadoop isn't so much speed but that it's been around and there is a pretty impressive and fairly mature set of projects that comprises the Hadoop ecosystem, from Yarn to Hive, etc. There are still many issues to resolve, and this evolution will continue for decades to come. The TB sort benchmark is pretty useless to me - I am much more concerned with stability, a vibrant community (which means people,…
As pointed out in the article multiple times, we are comparing with MR here. We are not comparing with Hadoop as an ecosystem. Spark plays nicely with Hadoop. As a matter of fact, this experiment ran on HDFS.
In terms of vibrant community, Spark is now the largest open source Big Data project by community/contributor count. More than 300 people have contributed code to the project.
Earlier quoted context omitted.
Since each node was handling 500GB of data (roughly), I think the disk speed may have been a more critical factor since each node had 244GB of memory. Their nodes used SSDs; the older nodes used spinning rust. The seek times alone will be a killer.
Not sure why you mentioned seek time. In large scale, distributed sorting, I/O is mostly sequential.
Earlier quoted context omitted.
I read that. But how does that compare with the nodes they're comparing against ("10x fewer nodes")?
The old entry had 10Gb/s (40 nodes/rack 160Gbps rack to spine. 2.5:1 subscription), 64GB of RAM, and 12 x 3TB SATA. The network part is probably the most important one here, and both have comparable network.
The strength of Hadoop isn't so much speed but that it's been around and there is a pretty impressive and fairly mature set of projects that comprises the Hadoop ecosystem, from Yarn to Hive, etc. There are still many issues to resolve, and this evolution will continue for decades to come. The TB sort benchmark is pretty useless to me - I am much more concerned with stability, a vibrant community (which means people,…
Actually Doug Cutting himself (who created Hadoop) tweeted about this. I guess Spark gets some of his blessing :) As pointed out in the article multiple times, we are comparing with MR here. We are not comparing with Hadoop as an ecosystem. Spark plays nicely with Hadoop. As a matter of fact, this experiment ran on HDFS. In terms of vibrant community, Spark is now the largest open source Big Data project by community…
Earlier quoted context omitted.
Not sure why you mentioned seek time. In large scale, distributed sorting, I/O is mostly sequential.
If only that were true -- the shuffle is typically seek-bound when the intermediate data doesn't fit into cache (plenty of papers show this pretty conclusively).
Except in the case of MR 2100 nodes the entire dataset fit in memory :)