Zero downtime Postgres migration, done right • Hacker News Board

Re: Zero downtime Postgres migration, done right

#11

Post by silviogutierrez » Fri, Jun 11, 2021, 4:48 PM UTC

fullstop wrote: post #8

silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

I work for a small company and it would be devastating if our database was down for a few hours.

Unplanned? Probably. But maybe it's not as a dire as you think? Azure Active Directory (auth as a service) went down for a while, globally, and life went on. Same with GCP and occasionally AWS[1]

I'm not saying there's no downside, I'm asking against the downside of engineering cost. And that itself carries risk of failure when you go live. It's not guaranteed.

[1] AWS is so ubiquitous though, that half the internet would be affected so it makes it less individually noticeable.

Re: Zero downtime Postgres migration, done right

#12

Post by jbverschoor » Fri, Jun 11, 2021, 4:49 PM UTC

fullstop wrote: post #8

silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

I work for a small company and it would be devastating if our database was down for a few hours.

I doubt that.. aws is down sometimes too. Card payments are sometimes down, even 112(911) is sometimes down.

I hope you have everything redundant, including the CEO

Re: Zero downtime Postgres migration, done right

#13

Post by hobs » Fri, Jun 11, 2021, 4:50 PM UTC

fullstop wrote: post #8

silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

I work for a small company and it would be devastating if our database was down for a few hours.

OTOH I worked for several small companies that had "business hours" software - if you had to take downtime on the weekend or after hours you'd be looking at impacting <1% of users.

Re: Zero downtime Postgres migration, done right

#14

Post by willcipriano » Fri, Jun 11, 2021, 4:50 PM UTC

fullstop wrote: post #8

silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

I work for a small company and it would be devastating if our database was down for a few hours.

In my view that's just saying at some point in the future the company will be devastated. Nothing has 100% uptime.

Re: Zero downtime Postgres migration, done right

#15

Post by chefkoch » Fri, Jun 11, 2021, 4:54 PM UTC

fullstop wrote: post #8

silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

I work for a small company and it would be devastating if our database was down for a few hours.

Then hopefully everything is HA in your company.

Re: Zero downtime Postgres migration, done right

#16

Post by jbverschoor » Fri, Jun 11, 2021, 4:54 PM UTC

> Blueground is a real estate tech company offering flexible and move-in ready furnished apartments across three continents and 12 of the world’s top cities. We search high and low for the best properties in the best cities, then our in-house design team transforms these spaces into turnkey spaces for 30 days or longer.

Seriously, how big can that db be, and how bad would a 1hr reduced availability / downtime be?

Seems like a lot of wasted engineering effort. “You are not google”

Re: Zero downtime Postgres migration, done right

#17

Post by ckboii89 » Fri, Jun 11, 2021, 4:55 PM UTC

does anyone know if this works when the target database is a replica/standby? The downside using pg_dump is that it acquires a lock on the table its dumping, and doing this on production may cause some slowness

Re: Zero downtime Postgres migration, done right

#18

Post by fullstop » Fri, Jun 11, 2021, 4:58 PM UTC

jbverschoor wrote: post #12

fullstop wrote: post #8

Earlier quoted context omitted.

I work for a small company and it would be devastating if our database was down for a few hours.

I doubt that.. aws is down sometimes too. Card payments are sometimes down, even 112(911) is sometimes down. I hope you have everything redundant, including the CEO

The database has a hot standby and two warm standby instances, redundant hardware, network infrastructure, all of that.

Re: Zero downtime Postgres migration, done right

#19

Post by cryptoz » Fri, Jun 11, 2021, 4:59 PM UTC

jbverschoor wrote: post #16

> Blueground is a real estate tech company offering flexible and move-in ready furnished apartments across three continents and 12 of the world’s top cities. We search high and low for the best properties in the best cities, then our in-house design team transforms these spaces into turnkey spaces for 30 days or longer. Seriously, how big can that db be, and how bad would a 1hr reduced availability / downtime be? See…

You cut out the part in About where they hint at it being important here even. "But we go a step further, merging incredible on-the-ground support with app-based services for a seamless experience for all our guests"

It sounds like across multiple timezones, with a tech stack that backs the business in a specific way, that downtime could be a problem and that it would reduce their offering, "seamless" and reliability for moving.

If twitter goes down and you can't tweet, only really Twitter loses money and you can try again later. But if you're moving into an apartment you don't want to be standing outside trying to get the keys but the system is down. Edit: And you also don't want the service to tell you that you can't move in from 11am-1pm because of 'maintenance'

Re: Zero downtime Postgres migration, done right

#20

Post by jlmorton » Fri, Jun 11, 2021, 4:59 PM UTC

If you are in AWS, or have connectivity available, AWS Database Migration Service makes this relatively trivial.

DMS for Postgres is based on Postgres Logical Replication, which is built-in to Postgres, and the same thing Bucardo is using behind the scenes. But AWS DMS is very nearly point-and-click to do this sort of migration.