21–30 of 46 posts
Comparing a column-oriented RDBMS with parallel query execution versus hadoop is a joke in the first place. Hadoop is extremely slow. That's nothing new. This is not an apples-to-apples comparison whatsoever. How does it compare against Greenplum or Aster or Vertica and is it more cost-effective? Those are important questions.
I am still new to large data, but isn't a solution like Redshift similar to Google's Big Query in that it only works with data that has a schema? How might one use Redshift with a db thats originally in Mongo?
Of course, if your data set can fit in memory, then Redshift or similar technologies probably is a better choice than Hive. But it's important to remember that the performance gains here come as the result of a tradeoff.
Disclaimer: I'm a committer on the Apache Hive project. A couple points in no particular order: * EMR Hive is a closed source fork of the upstream Apache Hive code base. The EMR docs imply that the latest version of EMR Hive is based on Apache Hive 0.8.1 (which was released more than a year ago), which means EMR users aren't benefitting from the performance improvements that appeared in the 0.9 and 0.10 releases. * I…
Disclaimer: I'm a committer on the Apache Hive project. A couple points in no particular order: * EMR Hive is a closed source fork of the upstream Apache Hive code base. The EMR docs imply that the latest version of EMR Hive is based on Apache Hive 0.8.1 (which was released more than a year ago), which means EMR users aren't benefitting from the performance improvements that appeared in the 0.9 and 0.10 releases. * I…
Because then the stupid headline wouldn't be so sensationalist, would it?
// I have no dog in this fight, but hate twisted claims
I wish the post had gone into depth on _why_ Redshift was significantly faster, but I'm betting it uses in-memory joins whereas (hence the size limitations it mentions) whereas Hive joins are just MapReduce jobs that keep only minimal subsets of data in memory at a given point. The upshot is the Hive/MapReduce strategy isn't limited by physical memory. Of course, if your data set can fit in memory, then Redshift or s…
The comparison is complete and utter joke.
I am still new to large data, but isn't a solution like Redshift similar to Google's Big Query in that it only works with data that has a schema? How might one use Redshift with a db thats originally in Mongo?
You won't fit that much data into Mongo anyways, so does it matter?
So I guess it does matter.
vs 1491 Hadoop
Looks like to me Hadoop is about 40 times faster...
So redshift took 155 seconds + 17 hours (17 * 3600) = 61355 secs total vs 1491 Hadoop Looks like to me Hadoop is about 40 times faster...