Redshift Performance and Cost
nerds.airbnb.com
Redshift Performance and Cost
1–10 of 47 posts
Re: Redshift Performance and Cost
#2A much better comparison would be a self hosted properly tuned PostgreSQL instance vs RedShift. :)
Re: Redshift Performance and Cost
#3This is the second article I've seen where the authors forget to multiply by the number of redshift nodes. A single XL node is $0.85/hr so 16 nodes would be $13.60/hr. Still cheaper than their Hive configuration obviously but less than a buck?
Re: Redshift Performance and Cost
#4Still running on posterous. Weird to think that this is probably the last posterous blog post I will read.
Re: Redshift Performance and Cost
#5Redshift is based on ParAccel, not on Postgres. ParAccel uses APIs similar to Postgres due to historical reasons, but not the technology.
For a basic overview: http://en.wikipedia.org/wiki/Paraccel
As for the rest of the article, it feels like a basic Data Warehousing 101 re-discovered. It should have been titled "Analytics: Back To The Future" :-)
Re: Redshift Performance and Cost
#6Redshift is based on ParAccel, not on Postgres. ParAccel uses APIs similar to Postgres due to historical reasons, but not the technology. For a basic overview: http://en.wikipedia.org/wiki/Paraccel As for the rest of the article, it feels like a basic Data Warehousing 101 re-discovered. It should have been titled "Analytics: Back To The Future" :-)
No kidding. The amount of startups that have flocked to hadoop for "data analytics" over the past 5 years is extremely disheartening. Almost all of the cases are far more suitable for any off-the-shelf RDBMS much less a column-oriented one. Same thing with MongoDB.
How much time and money would have been saved learning Database Theory/SQL/Data Warehousing/Dimensional Modeling instead of cramming everything into an unstructured data-store?
Re: Redshift Performance and Cost
#7The article mentions this briefly, but it should be emphasized: parallel loading from S3 is MUCH faster.
This weekend I loaded 2 billion rows from S3 both ways:
- From a single gzipped object: 4 hours 42 minutes
- From 2000 gzipped slices of 1M rows each: 17 minutes
(Loading from gzipped files is considerably faster, in addition to saving S3 charges.)
The article notes that choice of distribution key is critical. I'd add that choice of sort key is equally important. In my testing, a better sort key improved compression from 1.5:1 to 4:1, and also made common queries 5x faster.
Unfortunately, you only get one dist key and one sort key per table, so less common queries could get slower.
Re: Redshift Performance and Cost
#8Which storage format did you use for Hive? This is very important to how performance plays out, are you using snappy or LZO compression? Also, this is a relevant comment from a Hive committer (http://news.ycombinator.com/item?id=5248485).
Re: Redshift Performance and Cost
#9Redshift is based on ParAccel, not on Postgres. ParAccel uses APIs similar to Postgres due to historical reasons, but not the technology. For a basic overview: http://en.wikipedia.org/wiki/Paraccel As for the rest of the article, it feels like a basic Data Warehousing 101 re-discovered. It should have been titled "Analytics: Back To The Future" :-)
No kidding. The amount of startups that have flocked to hadoop for "data analytics" over the past 5 years is extremely disheartening. Almost all of the cases are far more suitable for any off-the-shelf RDBMS much less a column-oriented one. Same thing with MongoDB. How much time and money would have been saved learning Database Theory/SQL/Data Warehousing/Dimensional Modeling instead of cramming everything into an un…
Which off-the-shell RDBMS can handle queries over 3 billion rows?
Re: Redshift Performance and Cost
#10Earlier quoted context omitted.
No kidding. The amount of startups that have flocked to hadoop for "data analytics" over the past 5 years is extremely disheartening. Almost all of the cases are far more suitable for any off-the-shelf RDBMS much less a column-oriented one. Same thing with MongoDB. How much time and money would have been saved learning Database Theory/SQL/Data Warehousing/Dimensional Modeling instead of cramming everything into an un…
Which off-the-shell RDBMS can handle queries over 3 billion rows?
postgres?