Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Redshift Performance and Cost

nerds.airbnb.com

Post reply on HN
Print view View original

21–30 of 47 posts

Re: Redshift Performance and Cost

#21

Post by taligent »

meritt wrote: post #6
TY wrote: post #5

Redshift is based on ParAccel, not on Postgres. ParAccel uses APIs similar to Postgres due to historical reasons, but not the technology. For a basic overview: http://en.wikipedia.org/wiki/Paraccel As for the rest of the article, it feels like a basic Data Warehousing 101 re-discovered. It should have been titled "Analytics: Back To The Future" :-)

No kidding. The amount of startups that have flocked to hadoop for "data analytics" over the past 5 years is extremely disheartening. Almost all of the cases are far more suitable for any off-the-shelf RDBMS much less a column-oriented one. Same thing with MongoDB. How much time and money would have been saved learning Database Theory/SQL/Data Warehousing/Dimensional Modeling instead of cramming everything into an un…

Seriously can you and your ilk just please stop.

It's so exhausting to hear how much smarter you are and if we just educated ourselves we would realise the error of our ways. People who choose the technologies aren't stupid or masochistic. They understand their use case and the fact is that there are plenty of situations where SQL is suboptimal.

Re: Redshift Performance and Cost

#22

Post by taligent »

mixedbit wrote: post #9
meritt wrote: post #6

Earlier quoted context omitted.

No kidding. The amount of startups that have flocked to hadoop for "data analytics" over the past 5 years is extremely disheartening. Almost all of the cases are far more suitable for any off-the-shelf RDBMS much less a column-oriented one. Same thing with MongoDB. How much time and money would have been saved learning Database Theory/SQL/Data Warehousing/Dimensional Modeling instead of cramming everything into an un…

Which off-the-shell RDBMS can handle queries over 3 billion rows?

SAP HANA would be one but it is basically in memory so very, vey expensive.

Re: Redshift Performance and Cost

#23

Post by mallipeddi »

serbaut wrote: post #19

The first query seems awfully slow. I have a six node vertica cluster with a 100 column table with 7Bn rows in it and a similar query takes less than 3 seconds.

Disclosure: I work on the Redshift team.

The OP's cluster is a 16-node hs1.xlarge cluster (has 3 spindles per node). There's actually a more powerful node-type hs1.8xlarge which has 24 spindles on each node. More info: http://aws.amazon.com/redshift/pricing/

So it's not fair to compare Redshift performance to your Vertica cluster unless the hardware is similar.

Re: Redshift Performance and Cost

#24

Post by alexatkeplar »

nieksand wrote: post #12
meritt wrote: post #6

Earlier quoted context omitted.

No kidding. The amount of startups that have flocked to hadoop for "data analytics" over the past 5 years is extremely disheartening. Almost all of the cases are far more suitable for any off-the-shelf RDBMS much less a column-oriented one. Same thing with MongoDB. How much time and money would have been saved learning Database Theory/SQL/Data Warehousing/Dimensional Modeling instead of cramming everything into an un…

I think part of the issue why so many people have gone with Hive is that good, production-ready column stores are expensive. Redshift is posed to change that. If you're shopping in this space, Infobright is also worth checking out. And even for moderate data sizes (10+ GB per table), row store DBs tend to become painful. This is especially true when you need to support ad-hoc reporting queries, since the usual techni…

We use Infobright at SnowPlow (https://github.com/snowplow/snowplow), and are currently working on our Redshift integration.

One thing to be aware about with both is the lack of any support for wide tables - Infobright inherits MySQL's limit of 65,535 bytes per row (and UTF8 means 3 bytes per char); with Redshift you can stored wider rows but you can't query them (http://docs.aws.amazon.com/redshift/latest/dg/r_CREATE_TABLE...). Obviously not a deal breaker, but it locks you firmly in the densely populated rows, relational mindset.

This aside, we're super-excited about Redshift!

Re: Redshift Performance and Cost

#25

Post by PanMan »

meritt wrote: post #13
mixedbit wrote: post #9

Earlier quoted context omitted.

Which off-the-shell RDBMS can handle queries over 3 billion rows?

Counter-question: Which startup has a actual data table with over 3 billion rows?

We have just crossed 2 billion items in our datastore. While not 3 billion yet, I expect that to happen later this year.

Too bad Redshift can't handle JSON files: Converting everything will be annoying.

Re: Redshift Performance and Cost

#26

Post by mbesto »

taligent wrote: post #22
mixedbit wrote: post #9

Earlier quoted context omitted.

Which off-the-shell RDBMS can handle queries over 3 billion rows?

SAP HANA would be one but it is basically in memory so very, vey expensive.

And it's not a RDBMS. It's basically the same technology as RedShift, but not cloud based (yet).

Re: Redshift Performance and Cost

#27

Post by res0nat0r »

mbesto wrote: post #26
taligent wrote: post #22

Earlier quoted context omitted.

SAP HANA would be one but it is basically in memory so very, vey expensive.

And it's not a RDBMS. It's basically the same technology as RedShift, but not cloud based (yet).

Actually HANA One is available in the AWS Marketplace: https://aws.amazon.com/marketplace/pp/B009KA3CRY/ref=mkt_ste...

Re: Redshift Performance and Cost

#29

Post by jacques_chester »

mixedbit wrote: post #9
meritt wrote: post #6

Earlier quoted context omitted.

No kidding. The amount of startups that have flocked to hadoop for "data analytics" over the past 5 years is extremely disheartening. Almost all of the cases are far more suitable for any off-the-shelf RDBMS much less a column-oriented one. Same thing with MongoDB. How much time and money would have been saved learning Database Theory/SQL/Data Warehousing/Dimensional Modeling instead of cramming everything into an un…

Which off-the-shell RDBMS can handle queries over 3 billion rows?

In 2007 I worked for a firm with a 4 billion row join table in PostgreSQL. Might've been 7 or 8, I don't recall which. It ran on a quad core server with 16Gb of RAM. Joins going through this table took about 2-3 seconds to complete.
Post reply on HN