Live data from Hacker News

Amazon RedShift vs. local PostgreSQL

rarestblog.com

11–20 of 34 posts

Re: Amazon RedShift vs. local PostgreSQL

#11
post #4

I'd like to see some more information about the local setup, including hardware and the postgresql.conf. Otherwise, this tells me very little in terms of comparison.

In anyway this test won't tell you much, just how different systems behave to bigger load. The local setup was quite usual: PostgreSQL 9.2, Mint 13, default conf in VirtualBox in iMac i5 12GB. (read: home computer, no tuning) For me the result is that mostly RedShift is on par with local PostgreSQL, sometimes even winning for Also the big deal was that RedShift scaled linearly.

It scaled linearly, but also went unresponsive for five minutes. (Yeah yeah, it's a new service).

The default Postgres configuration is pretty weak. work_mem is set way to low, for instance, and that's bitten me a few times. I wouldn't say it's unrealistic--lots of people run with it that way in production and never find out how easily they could speed things up. Even me, for years.

But ultimately I'm more swayed by your interaction with it and I hate the endless benchmark tweaking that comes after every blog post about performance testing stuff. The point of this Redshift thing is hugeness first and foremost, so it's interesting.

Re: Amazon RedShift vs. local PostgreSQL

#13

Wouldn't it have been better to do an EXPLAIN ANALYZE for the timing measurements instead of having the results returned locally?

Honestly I don't know. Do you think it would be better? Could you explain why?

because it shows the estimated cost (plan) of the queries independent of the magically subtracted network roundtrip time. it also serves to show if the redshift even supports it.it.

Re: Amazon RedShift vs. local PostgreSQL

#16

very interesting. one of the reasons we picked mysql for a very high-volume app over postgres is that we have RDS and didn't want to do backups/snapshots/etc. Could we now use RedShift as a postgres-API RDS?

You wouldn't want to replace RDS (or MySQL/postgres) with Redshift for OLTP workloads. Redshift is built for analytics and complex query workloads. The major benefit is the ability to do MPP (Massive Parallel Processing) and distribute the work out to nodes. I believe you can have up to 100 nodes in the cluster with the 8XL machines and 32 with the single XL.

Re: Amazon RedShift vs. local PostgreSQL

#17

very interesting. one of the reasons we picked mysql for a very high-volume app over postgres is that we have RDS and didn't want to do backups/snapshots/etc. Could we now use RedShift as a postgres-API RDS?

RedShift is tuned for analytics, so if you're using MySQL today for analytics, it could be a good fit. But if you're looking for more of an OLTP engine, RedShift is probably not the right choice for you.

Re: Amazon RedShift vs. local PostgreSQL

#18

Earlier quoted context omitted.

How big is your data?

16tb per instance

Cool. :)

I wouldn't know a better way to deal with that with Postgres than sharding across instances. But my "big data" is about 0.75 TB so it fits nicely in one instance. I don't know how people with real problems do it.

Re: Amazon RedShift vs. local PostgreSQL

#19
post #13

Earlier quoted context omitted.

Honestly I don't know. Do you think it would be better? Could you explain why?

because it shows the estimated cost (plan) of the queries independent of the magically subtracted network roundtrip time. it also serves to show if the redshift even supports it.it.

Ahhh, you meant on the RedShift side. I did not think of it. It would have definitely been better. Good point!

It seems that it actually supports it http://docs.aws.amazon.com/redshift/latest/dg/r_EXPLAIN.html

Re: Amazon RedShift vs. local PostgreSQL

#20
I really don't understand what's going on here.

* You're measuring request latency. What part of that (for RedShift) is due to the network? (EDIT: I re-read and saw you're using `SELECT 1` as a gauge for round-trip latency and subtracting it from the results. Are you only doing this for RedShift, or also for local PostgreSQL? To me, it seems like that heuristic is over broad -- it encapsulates not only network latency, but syscall overhead, query parsing, etc).

* In your tests, PostgreSQL without indices performs on-par with RedShift. Does RedShift not support indexing? Is there some metric you're trying to show by not using indices? As designed, this benchmark does not map to any use-case I've ever seen.

Post reply on HN