Amazon RedShift vs. local PostgreSQL • Hacker News Board

Re: Amazon RedShift vs. local PostgreSQL

#31

Post by ramarnat » Sat, Feb 16, 2013, 11:18 AM UTC

rarestblog wrote: post #5

csummers wrote: post #4

I'd like to see some more information about the local setup, including hardware and the postgresql.conf. Otherwise, this tells me very little in terms of comparison.

In anyway this test won't tell you much, just how different systems behave to bigger load. The local setup was quite usual: PostgreSQL 9.2, Mint 13, default conf in VirtualBox in iMac i5 12GB. (read: home computer, no tuning) For me the result is that mostly RedShift is on par with local PostgreSQL, sometimes even winning for Also the big deal was that RedShift scaled linearly.

Like others have mentioned, Postgres and Redshift are very different animals, Postgres is a row store and Redshift is a column store. On large data sets analytic queries that return a few columns will significantly outperform a row store DB.

We have found that Redshift is comparable to other columnar databases we work with, while we cannot publish any comparative benchmarks, we did put a blog post on what we found (link in another comment here)

Re: Amazon RedShift vs. local PostgreSQL

#32

Post by ramarnat » Sat, Feb 16, 2013, 11:18 AM UTC

rarestblog wrote: post #5

csummers wrote: post #4

I'd like to see some more information about the local setup, including hardware and the postgresql.conf. Otherwise, this tells me very little in terms of comparison.

In anyway this test won't tell you much, just how different systems behave to bigger load. The local setup was quite usual: PostgreSQL 9.2, Mint 13, default conf in VirtualBox in iMac i5 12GB. (read: home computer, no tuning) For me the result is that mostly RedShift is on par with local PostgreSQL, sometimes even winning for Also the big deal was that RedShift scaled linearly.

Like others have mentioned, Postgres and Redshift are very different animals, Postgres is a row store and Redshift is a column store. On large data sets analytic queries that return a few columns will significantly outperform a row store DB.

We have found that Redshift is comparable to other columnar databases we work with, while we cannot publish any comparative benchmarks, we did put a blog post on what we found (link in another comment here)

Re: Amazon RedShift vs. local PostgreSQL

#33

Post by monstrado » Sat, Feb 16, 2013, 7:38 PM UTC

eksith wrote: post #25

iblaine wrote: post #22

What this test is essentially doing is comparing Postgres against a single node of Redshift. It is not surprising that Postgres is faster. But Redshift is not meant to be used on a single node. What Postgres & Redshift represent are are two different products for two very different problems. Postgres is good for small sets of transactional data like orders in a shopping cart system (less than 1TB). Redshift is good f…

We don't run a shopping cart, but one of our databases at present is at 11.3TB on PostgreSQL 9.1 and we're by no means dealing with small sets. We routinely juggle several Gigs at a time when we need to do analytics. We didn't see a reason to put this on a cloud since bandwidth + electricity is still cheaper for us than bandwidth + storage in the cloud at present.

If you have a few servers to spare, I'd recommend installing Cloudera Impala on them. You can use Apache Sqoop to pull the data out of Postgres and into HDFS.. Directly after, you can run SQL queries which will query the data in parallel (similar to redshift).

Re: Amazon RedShift vs. local PostgreSQL

#34

Post by scw » Mon, Feb 25, 2013, 9:53 PM UTC

zwily wrote: post #27

scotth wrote: post #24

Earlier quoted context omitted.

Redshift isn't Postgres. It's built on top of Postgres, similar to how Infobright is built on top of MySQL.

Redshift is built on ParAccel, not Postgres. It speaks the Postgres API, but the storage engine is not Postgres.

Some of the code is based on Postgres, based on their own marketing materials: "Paraccel has leveraged Postgres for some of its parsing and planning functions". So the head node continues to have Postgres origins, but not the compute nodes. Also see this write-up on their paper discussing the links to Postgres: http://dbmsmusings.blogspot.com/2009/07/paraccel-and-their-p...