Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon Redshift is 10x faster and cheaper than Hadoop and Hive

slideshare.net

Post reply on HN
Print view View original

41–46 of 46 posts

Re: Amazon Redshift is 10x faster and cheaper than Hadoop and Hive

#41

Post by seanmcdirmid »

MBCook wrote: post #40
free652 wrote: post #29

So redshift took 155 seconds + 17 hours (17 * 3600) = 61355 secs total vs 1491 Hadoop Looks like to me Hadoop is about 40 times faster...

That's like saying a bicycle is faster than a car because I can buy a bike in 10 minutes while it may take me a couple of hours to get through the car's paperwork. If you do enough queries, redshift will come out faster (assuming the numbers are correct).

That's assuming Hive doesn't have it's own special format that could be converted to to improve performance, right?

Re: Amazon Redshift is 10x faster and cheaper than Hadoop and Hive

#42

Post by TallGuyShort »

MBCook wrote: post #40
free652 wrote: post #29

So redshift took 155 seconds + 17 hours (17 * 3600) = 61355 secs total vs 1491 Hadoop Looks like to me Hadoop is about 40 times faster...

That's like saying a bicycle is faster than a car because I can buy a bike in 10 minutes while it may take me a couple of hours to get through the car's paperwork. If you do enough queries, redshift will come out faster (assuming the numbers are correct).

If you do enough queries, you should spend the time to use RCFile for Hive, in which case redshift wont come out _that_ much faster. The point is the 17 hours is not negligible.

Re: Amazon Redshift is 10x faster and cheaper than Hadoop and Hive

#43

Post by dromidas »

TallGuyShort wrote: post #42
MBCook wrote: post #40

Earlier quoted context omitted.

That's like saying a bicycle is faster than a car because I can buy a bike in 10 minutes while it may take me a couple of hours to get through the car's paperwork. If you do enough queries, redshift will come out faster (assuming the numbers are correct).

If you do enough queries, you should spend the time to use RCFile for Hive, in which case redshift wont come out _that_ much faster. The point is the 17 hours is not negligible.

That is a good case since customers who typically need a datawarehouse aren't just going to upload data once... they probably are going to upload frequently.

Re: Amazon Redshift is 10x faster and cheaper than Hadoop and Hive

#44

Post by BrianEatWorld »

scotth wrote: post #12
BrianEatWorld wrote: post #9

I am still new to large data, but isn't a solution like Redshift similar to Google's Big Query in that it only works with data that has a schema? How might one use Redshift with a db thats originally in Mongo?

Impose a schema.

Is there an easy way to go back and forth between data with schema and data without? I'd love the benefits of this for queries, but for the production side of things, imposing a schema would be costly.

Re: Amazon Redshift is 10x faster and cheaper than Hadoop and Hive

#45

Post by TallGuyShort »

dromidas wrote: post #43
TallGuyShort wrote: post #42

Earlier quoted context omitted.

If you do enough queries, you should spend the time to use RCFile for Hive, in which case redshift wont come out _that_ much faster. The point is the 17 hours is not negligible.

That is a good case since customers who typically need a datawarehouse aren't just going to upload data once... they probably are going to upload frequently.

You're missing my point and resorting to sarcasm - very nice . My point is not that Hive is the better choice because everyone is going to reload their data frequently. My point is that if you want a fair benchmark, don't use an obviously slow data format for Hive. They spent time importing data optimized for RedShift, but they took a very naive approach for Hive. I'm sure RedShift will still be faster, but not 10 times faster.
Post reply on HN