Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

thecodinginterface.com

Post reply on HN
Print view View original

21–30 of 40 posts

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#21

Post by ekump »

This is a great write-up! I wrote a far less thorough blog post on the same topic a few months ago: https://vericred.github.io/2020/08/10/migrating-to-aurora.ht....

Overall, we're happy with our decision to migrate to Aurora. We feel it offers better performance, observability and scalability compared to heroku postgres. To echo some other comments:

1. Jesse at Heroku is great. I had never setup my own postgres instance before, and his guidance helped me a lot.

2. We used DMS, and it was ok, not great. Dealing with LOBs was a bit of a pain. In order to keep the migration time to a minimum we setup multiple DMS tasks and grouped tables into those tasks based on their max LOB size. Most of our tables didn't have any LOB columns and they migrated quickly.

3. The ability to fast clone a database is nice. We can create a copy of our production database in about five minutes for use in non-prod environments. This would take a couple of hours in heroku.

4. The Aurora IO costs are ridiculous. Are there any good tools to see what is exactly causing so much I/O?

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#22

Post by mnutt »

hyperdimension wrote: post #17
mnutt wrote: post #11

Earlier quoted context omitted.

Aurora (Postgres) looked like it would be great, and the performance for our use case was much better than RDS. But there was a showstopper for us with the replication: If the cluster leader fails, _every read replica will force a restart_ to elect a new leader, resulting in minimum ~30s complete cluster downtime. In our situation we have no problem with a short downtime in write availability, but the fact that we ca…

I'm no DBA, but is that not the entire point of a cluster/replicas? Sure, maybe writes won't be accepted or replicated until a new master is elected, but you can't even read from the cluster? I'm curious at this point: what is it exactly that you gain from a cluster? Is it only having to wait 30s vs. however long it takes for the master to come back online?

That was the mind-boggling part to me. After testing failover and noticing the behavior we had a number of discussions with AWS reps and support engineers that confirmed it. And Aurora's failover documentation hints at it but downplays the severity.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#23

Post by mijoharas »

yowlingcat wrote: post #19
mijoharas wrote: post #5

Might have been nice to see this post a year ago :). My company migrated our ~2TB postgres db from heroku to AWS (originally aurora, we switched back to RDS postgres). Haven't had time for a blog post, but figure I might post our learnings here, and the differences in our approach. * We did the second step (ec2 -> aurora) via AWS DMS (database migration service). This service may be a little more of a pain than it's…

Fantastic comment, thank you so much. I've been following Aurora for a while but a little curious (and skeptical) about sharp edges vs vanilla PG. If I may ask you a follow-on question, who do you think Aurora would be the right fit for?

Hmmmm... That's a good question.

The performance was great, I can't complain about it, but the unpredictable (and high) price was the issue.

In terms of aurora I think it probably can scale better to much larger workloads. If we put the time in and dug in and optimised any queries that could touch a large number of blocks (we do well, but I'm sure there are queries that have some seq scans somewhere in our codebase) then the price wouldn't have been an issue, and we'd happily have stayed with it.

In addition, we have a single large master and one replica. with my understanding of aurora's architecture, and how the storage is separated from the "postgres" instance, it could possibly do a lot better with drastically different architectures with a lot of different smaller "postgres" instances. We didn't have time to investigate many other architectures and in the end moving to RDS felt safest (didn't want to spend too much time when we could be building core features)

Hope that answers your question! Some of the above is more an educated guess with a smattering of experience so please treat it as such.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#24

Post by mijoharas »

mijoharas wrote: post #23
yowlingcat wrote: post #19

Earlier quoted context omitted.

Fantastic comment, thank you so much. I've been following Aurora for a while but a little curious (and skeptical) about sharp edges vs vanilla PG. If I may ask you a follow-on question, who do you think Aurora would be the right fit for?

Hmmmm... That's a good question. The performance was great, I can't complain about it, but the unpredictable (and high) price was the issue. In terms of aurora I think it probably can scale better to much larger workloads. If we put the time in and dug in and optimised any queries that could touch a large number of blocks (we do well, but I'm sure there are queries that have some seq scans somewhere in our codebase)…

On the other hand, vanilla postgres is very powerful, and you shouldn't underestimate how much you can win by cranking up the PIOPs for an RDS postgres instance.

Remember, always benchmark representative workloads.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#25

Post by shangxiao »

Nice!

Slight nitpick: Django should automatically create the Hstore extension for you [1] during migrate whenever you use a HStoreField

[1] https://github.com/django/django/blob/f5e07601b233a50e6bcca4...

Edit: I might've been mistaken here in thinking the above migration operation was automatically applied as the docs seem to suggest you need to add it yourself [2] but at least it's as easy as adding an operation ;)

[2] https://docs.djangoproject.com/en/3.1/ref/contrib/postgres/f...

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#26

Post by mijoharas »

jrochkind1 wrote: post #16
pedrokost wrote: post #12

The author has gained a significant amount of Postgres-related knowledge when performing this migration. Taking this into consideration, is it still worth using the managed Aurora, instead of the EC2 self-managed instance?

RDS Postgres (actual postgres, not the Aurora product which can be "mostly postgres compatible") is another option, not just "EC2 self-managed instance". I guess we'd have to look at price comparisons for the size you actually need. I think depending on load, the price differences may be nominal.

Thought I'd weigh in here, since we moved from heroku to ec2 to aurora to RDS postgres, so I can probably speak to this a little more.

* ec2 self managed is easily the cheapest, we had a solid setup, with continuous backups and a read replica, if cost is a factor, it's easily a winner. However, there is a _lot_ of knowledge that goes with it. When it comes down to it, you can pay someone else to handle that. This isn't just the setup cost, you need to factor in ongoing maintenance (the number of people at my company that could have done complicated things with the instance was probably me and another engineer, and we didn't want to be permanently on call for this) and general risk.

* EC2 did probably work out to between 1/2 to 3/4 (probably 2/3) the price of the equivalent RDS (tough to say exactly, as I'd need to factor in all the ancillary costs that are more "bundled together" in RDS)

* RDS was much cheaper than aurora for our workload. (almost 1/2 the price).

I think the main thing to remember, is that ec2 is much cheaper than RDS, but RDS is cheaper than engineers. With that, while we didn't need to do anything complicated other than the migration, the risk and possible engineering time and bus factor didn't feel worth it to stay on ec2.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#27

Post by rohansahai »

mijoharas wrote: post #26
jrochkind1 wrote: post #16

Earlier quoted context omitted.

RDS Postgres (actual postgres, not the Aurora product which can be "mostly postgres compatible") is another option, not just "EC2 self-managed instance". I guess we'd have to look at price comparisons for the size you actually need. I think depending on load, the price differences may be nominal.

Thought I'd weigh in here, since we moved from heroku to ec2 to aurora to RDS postgres, so I can probably speak to this a little more. * ec2 self managed is easily the cheapest, we had a solid setup, with continuous backups and a read replica, if cost is a factor, it's easily a winner. However, there is a _lot_ of knowledge that goes with it. When it comes down to it, you can pay someone else to handle that. This isn…

Been on RDS for awhile now but have recently had some issues that are pushing us to heavily consider EC2 - especially since we've heard about considerable performance gains running on some of the bare metal EC2 options out there vs RDS.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#28

Post by rohansahai »

sciguymcq wrote: post #15
pedrokost wrote: post #12

The author has gained a significant amount of Postgres-related knowledge when performing this migration. Taking this into consideration, is it still worth using the managed Aurora, instead of the EC2 self-managed instance?

I currently manage, or am "lead dev", for around a half dozen to 10 apps using Postgres as the database. For two of those I use a managed database service (AWS RDS / AWS Aurora Postgres) because they drive mission critical high value products / services and I wouldn't consider using a standalone install of Postgres. For other apps that are not so mission critical I am perfectly happy saving considerable money and run…

That's super interesting! Have heard people do the opposite for performance reasons (run on bare EC2) and storage cost reason. Do you notice a big performance difference between the EC2 apps and RDS apps? Also have you had any EBS volume related issues on RDS or unexplained latency/downtime ?

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#29

Post by mijoharas »

rohansahai wrote: post #27
mijoharas wrote: post #26

Earlier quoted context omitted.

Thought I'd weigh in here, since we moved from heroku to ec2 to aurora to RDS postgres, so I can probably speak to this a little more. * ec2 self managed is easily the cheapest, we had a solid setup, with continuous backups and a read replica, if cost is a factor, it's easily a winner. However, there is a _lot_ of knowledge that goes with it. When it comes down to it, you can pay someone else to handle that. This isn…

Been on RDS for awhile now but have recently had some issues that are pushing us to heavily consider EC2 - especially since we've heard about considerable performance gains running on some of the bare metal EC2 options out there vs RDS.

We had equivalent performance (and we'd benchmarked both) for equivalent RDS and EC2 instances. (EDIT: to note, we didn't dig in heavily, just looked and saw it seemed the same, maybe there's something smart you can do to make EC2 a lot faster)

First thing to think about for RDS (or any postgres instance really) is to figure out what the bottlenecks are, if your cache hit ratio is slightly low, or queries are generally feeling a little sluggish but cpu and ram aren't too high, I'd recommend looking into increasing the provisioned IOPS of the machine (requires no downtime, basically you can just make the storage faster).

If your issues are ram/cpu being high you might want to consider moving to a larger instance.

It can also be worth looking into the number of temp files being created by your queries and tweak your work_mem setting (you can use log_temp_files to see if this is likely to help a bit).

Other than that, remember that every connection to the db is a separate process, so you don't want too many. It could be worth looking at RDS proxy or pg_bouncer if you think you have a lot of connections to your db.

Re: Migrating Large Heroku Postgres Instances to AWS Aurora Without Downtime

#30

Post by mijoharas »

ekump wrote: post #21

This is a great write-up! I wrote a far less thorough blog post on the same topic a few months ago: https://vericred.github.io/2020/08/10/migrating-to-aurora.ht... . Overall, we're happy with our decision to migrate to Aurora. We feel it offers better performance, observability and scalability compared to heroku postgres. To echo some other comments: 1. Jesse at Heroku is great. I had never setup my own postgres inst…

Hey, regarding 4. we came across this but didn't find anything from the article particularly helpful for us:

https://aws.amazon.com/blogs/database/reducing-aurora-postgr...

(we bounced to normal RDS postgres and reduced our costs a lot while keeping performance similar, check out my other comments on this thread for details).

Post reply on HN