Amazon Redshift is 10x faster and cheaper than Hadoop and Hive

I haven't tried redshift before, but coming from a MR/Hadoop/Hive background, this seems to me like quite a sensational claim. I'd be very keen to hear other's thoughts on how widely these kinds of gains would apply for BigData processing.

As Carl Sagan said..

"Extraordinary claims require extraordinary evidence"

http://en.wikipedia.org/wiki/Carl_Sagan

Slide 2&6, one query every 30 minutes.

It turns out usage based billing can be cheaper if you don't use a resource.

I'm willing to bet that's a not-uncommon scenario for a lot of organizations, however. If you're doing continuous querying of large amounts of data, then it's probably worth building your own hadoop cluster (physically or via Amazon), but a lot of people are just going to accumulate data and then make queries against it. Lots of 'active users per day', 'traffic by hour', 'purchases by popularity', etc. only get run to create data for the CEO every morning, or by the marketing manager every afternoon, that sort of thing.

Hive is not particularly fast in and of itself; it just has horizontal scaling and a SQL-ish front-end. Looking at AWS RedShift's homepage[1] (emphasis added):

> Amazon Redshift delivers fast query and I/O performance for virtually any size dataset by using columnar storage technology and parallelizing and distributing queries across multiple nodes.

Column stores databases[2] can be screamingly fast for analytics operations compared to RDBMS or other DB types (ala assorted NoSQL). See Kdb[3] or MonetDB[4] for examples of specific implementations. I'd fully expect a competent column store designed for horizontal scaling to obliterate Hive for a wide range of problems.

The usual big-data caveat: you need to pay attention to the fit of your tools against your problem and your data. I don't expect RedShift to be any different. Still, it's pretty exciting to see a new analysis DB tech cropping up like this. And doubly interesting to see this coming from Amazon.

[1] https://aws.amazon.com/redshift/

[2] https://en.wikipedia.org/wiki/Column-oriented_DBMS

[3a] http://kx.com/kdb-plus.php

[3b] https://en.wikipedia.org/wiki/K_%28programming_language%29#K...

[4] http://www.monetdb.org/Home

Comparing a column-oriented RDBMS with parallel query execution versus hadoop is a joke in the first place. Hadoop is extremely slow. That's nothing new. This is not an apples-to-apples comparison whatsoever.

How does it compare against Greenplum or Aster or Vertica and is it more cost-effective? Those are important questions.