Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

thecodinginterface.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#31

Post by aeyes »

ekump wrote: post #21

This is a great write-up! I wrote a far less thorough blog post on the same topic a few months ago: https://vericred.github.io/2020/08/10/migrating-to-aurora.ht... . Overall, we're happy with our decision to migrate to Aurora. We feel it offers better performance, observability and scalability compared to heroku postgres. To echo some other comments: 1. Jesse at Heroku is great. I had never setup my own postgres inst…

> 4. The Aurora IO costs are ridiculous. Are there any good tools to see what is exactly causing so much I/O?

I don't use Aurora so I don't know what is available but pg_stat_statements does have I/O statistics so you can review it manually or use a tool like pgAnalyze (not sure if it supports Aurora) to see what queries are I/O intensive.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#32

Post by craigkerstiens »

ekump wrote: post #21

This is a great write-up! I wrote a far less thorough blog post on the same topic a few months ago: https://vericred.github.io/2020/08/10/migrating-to-aurora.ht... . Overall, we're happy with our decision to migrate to Aurora. We feel it offers better performance, observability and scalability compared to heroku postgres. To echo some other comments: 1. Jesse at Heroku is great. I had never setup my own postgres inst…

DMS under the covers is another tool. The Postgres support for it lags pretty well and is mostly and after thought. It's much better to try to leverage something more native, either the WAL or if on a more modern version take advantage of logical replication.

For coming off Heroku DMS isn't going to work cause you don't have a replication slot, so your only approach is wal-e, or maybe wal-g?

Fast cloning is one nice feature for sure of Aurora, but when you go into Aurora you're starting to deviate more and more from native Postgres. Fast cloning can definitely be done one without Aurora. We're working on bringing our ability on Crunchy Bridge down to a few minutes similar to Aurora, while still being 100% Postgres. Not there yet, but suspect we're a few months away.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#33

Post by craigkerstiens »

mijoharas wrote: post #26
jrochkind1 wrote: post #16

Earlier quoted context omitted.

RDS Postgres (actual postgres, not the Aurora product which can be "mostly postgres compatible") is another option, not just "EC2 self-managed instance". I guess we'd have to look at price comparisons for the size you actually need. I think depending on load, the price differences may be nominal.

Thought I'd weigh in here, since we moved from heroku to ec2 to aurora to RDS postgres, so I can probably speak to this a little more. * ec2 self managed is easily the cheapest, we had a solid setup, with continuous backups and a read replica, if cost is a factor, it's easily a winner. However, there is a _lot_ of knowledge that goes with it. When it comes down to it, you can pay someone else to handle that. This isn…

We've had a few people migrate from Heroku and RDS over to Crunchy Bridge and see 2.5x performance improvement on warm cache and up to 5-6x performance improvement on cold cache for the exact same $ spend. I can't say easily how that would compare to raw EC2 cause it really depends on how you tune and configure. But there is definitely some opportunity to optimize vs. any stock install and setup.

The real thing about EC2 is it'll be cheaper. But when you do have a bad day, say page corruption you're gonna have more than a bad day and a bad week/month trying to untangle that. I'm not sure how helpful RDS is in those cases. I can say that Heroku historically was solid (I was there way back in the day having helped build the service). And there are some other options that can still deliver good performance, good support, at a balanced price.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#34

Post by nijave »

ekump wrote: post #21

This is a great write-up! I wrote a far less thorough blog post on the same topic a few months ago: https://vericred.github.io/2020/08/10/migrating-to-aurora.ht... . Overall, we're happy with our decision to migrate to Aurora. We feel it offers better performance, observability and scalability compared to heroku postgres. To echo some other comments: 1. Jesse at Heroku is great. I had never setup my own postgres inst…

Not sure about cost and availability but Performance Insights for RDS postgres will show waits on various resources (disk, CPU) and is fairly easy to use. If it's like RDS postgres you can setup logging and use something like pgbadger (you would need to adjust the parameter group logging options to capture the right details)

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#35

Post by nijave »

mijoharas wrote: post #30
ekump wrote: post #21

This is a great write-up! I wrote a far less thorough blog post on the same topic a few months ago: https://vericred.github.io/2020/08/10/migrating-to-aurora.ht... . Overall, we're happy with our decision to migrate to Aurora. We feel it offers better performance, observability and scalability compared to heroku postgres. To echo some other comments: 1. Jesse at Heroku is great. I had never setup my own postgres inst…

Hey, regarding 4. we came across this but didn't find anything from the article particularly helpful for us: https://aws.amazon.com/blogs/database/reducing-aurora-postgr... (we bounced to normal RDS postgres and reduced our costs a lot while keeping performance similar, check out my other comments on this thread for details).

& with RDS postgres, in a lot of cases, you're probably o.k. with gp2 vs the much more expensive io1 provisioned IOPs storage unless you have a fairly business database (100s-1000s qps)

The main difference is 95% performance SLA on gp2 vs >99 on io1

(You get 3x IOP per GB on gp2)

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#36

Post by takeda »

pedrokost wrote: post #12

The author has gained a significant amount of Postgres-related knowledge when performing this migration. Taking this into consideration, is it still worth using the managed Aurora, instead of the EC2 self-managed instance?

If you have a CM infrastructure set up, you're not gaining much. Tools like repmgr/barman (in public cloud you probably would use WAL-E/WAL-G) provide nearly all of the benefits.

PostgreSQL is generally low maintenance, the only cases when this might not be true is when you're hitting performance bottlenecks and need to tune your database, optimize your tables, or use bigger instance etc, but RDS doesn't save you from that.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#37

Post by oblio »

mbesto wrote: post #13
tenaciousDaniel wrote: post #2

I'm considering moving from a self-hosted Mongo setup to Aurora, either with PG or MySQL. Anyone here have opinions on Aurora?

I've heard mixed reviews on Aurora MySQL. Some people move to it and see huge performance gains, others simply don't or have some limitations.

Do you have some examples of limitations they faced?

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#38

Post by base »

nijave wrote: post #35
mijoharas wrote: post #30

Earlier quoted context omitted.

Hey, regarding 4. we came across this but didn't find anything from the article particularly helpful for us: https://aws.amazon.com/blogs/database/reducing-aurora-postgr... (we bounced to normal RDS postgres and reduced our costs a lot while keeping performance similar, check out my other comments on this thread for details).

& with RDS postgres, in a lot of cases, you're probably o.k. with gp2 vs the much more expensive io1 provisioned IOPs storage unless you have a fairly business database (100s-1000s qps) The main difference is 95% performance SLA on gp2 vs >99 on io1 (You get 3x IOP per GB on gp2)

io1 also has a slight better performance than gp2

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#39

Post by aetherson »

hyperdimension wrote: post #17
mnutt wrote: post #11

Earlier quoted context omitted.

Aurora (Postgres) looked like it would be great, and the performance for our use case was much better than RDS. But there was a showstopper for us with the replication: If the cluster leader fails, _every read replica will force a restart_ to elect a new leader, resulting in minimum ~30s complete cluster downtime. In our situation we have no problem with a short downtime in write availability, but the fact that we ca…

I'm no DBA, but is that not the entire point of a cluster/replicas? Sure, maybe writes won't be accepted or replicated until a new master is elected, but you can't even read from the cluster? I'm curious at this point: what is it exactly that you gain from a cluster? Is it only having to wait 30s vs. however long it takes for the master to come back online?

The thing you gain from the replicas is removing read load from the primary during normal operations.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#40

Post by mbesto »

oblio wrote: post #37
mbesto wrote: post #13

Earlier quoted context omitted.

I've heard mixed reviews on Aurora MySQL. Some people move to it and see huge performance gains, others simply don't or have some limitations.

Do you have some examples of limitations they faced?

Honestly I don't sorry.
Post reply on HN