Zero downtime Postgres migration, done right

You can do multi master replication with conflicts detection with symmetricDS. The migration would be similar to the procedure described in the article.

Re: Zero downtime Postgres migration, done right

#6

Post by luhn » Fri, Jun 11, 2021, 4:31 PM UTC

Braintree (IIRC) had a really clever migration strategy, although I can't seem to find the blog post now. They paused all traffic at the load balancer, cut over to the new DB, and then resumed traffic. No requests failed, just a slight bump in latency while the LBs were paused.

This app apparently had robust enough retry mechanisms that they were able just eat the errors and not have customer issues—Color me impressed! I'm not sure how many teams can make that claim; that's a hard thing to nail down.

Re: Zero downtime Postgres migration, done right

#7

Post by silviogutierrez » Fri, Jun 11, 2021, 4:32 PM UTC

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad?

I looked at the company, and while they seem rather large, they're not Netflix or AWS.

I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be thought out and weighed against the engineering effort.

Don't get me wrong, this is very cool.

But I wonder what the engineering cost was. I'd think easily in the hundreds of thousands of dollars.

Re: Zero downtime Postgres migration, done right

#8

Post by fullstop » Fri, Jun 11, 2021, 4:34 PM UTC

silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

I work for a small company and it would be devastating if our database was down for a few hours.

Re: Zero downtime Postgres migration, done right

#9

Post by rhacker » Fri, Jun 11, 2021, 4:35 PM UTC

"the zero downtime" + this site is definitely down is making me laugh.

Re: Zero downtime Postgres migration, done right

#10

Post by scrollaway » Fri, Jun 11, 2021, 4:38 PM UTC

luhn wrote: post #6

Braintree (IIRC) had a really clever migration strategy, although I can't seem to find the blog post now. They paused all traffic at the load balancer, cut over to the new DB, and then resumed traffic. No requests failed, just a slight bump in latency while the LBs were paused. This app apparently had robust enough retry mechanisms that they were able just eat the errors and not have customer issues—Color me impresse…

I remember reading something like that about Adyen. That might be why you're unable to find it.