Live data from Hacker News

Zero downtime Postgres migration, done right

engineering.theblueground.com

21–30 of 135 posts

Re: Zero downtime Postgres migration, done right

#21
post #8

Earlier quoted context omitted.

I work for a small company and it would be devastating if our database was down for a few hours.

Then hopefully everything is HA in your company.

It absolutely is, BGP with geographically diverse paths, databases, app servers, etc, are all redundant. It's hosted in-house, so there is a cold standby database in AWS which would only be used if, say, an aircraft crashed into our server rooms.

Re: Zero downtime Postgres migration, done right

#23
post #6

Braintree (IIRC) had a really clever migration strategy, although I can't seem to find the blog post now. They paused all traffic at the load balancer, cut over to the new DB, and then resumed traffic. No requests failed, just a slight bump in latency while the LBs were paused. This app apparently had robust enough retry mechanisms that they were able just eat the errors and not have customer issues—Color me impresse…

I remember reading something like that about Adyen. That might be why you're unable to find it.

pgbouncer has PAUSE comnand, which can be used for seamless restarts/failovers or similar

Re: Zero downtime Postgres migration, done right

#24

I wonder how much easier software engineering would be if there were a period where things are simply not available. What problems are currently very difficult would be made trivial if 6 hours of downtime every Sunday were acceptable? 10PM-4AM EST

I find it very hard to come up with a use case where a weekly 6 hours of downtime at night EST would be acceptable.

US government websites already often do this, and I find that completely unacceptable since these services need to be available to anyone, regardless of work and life schedules.

Any website that's international would also suffer greatly for an EST centralized scheduled downtime.

Maybe a very localized website that doesn't have much impact on real life?

Re: Zero downtime Postgres migration, done right

#26

I wonder how much easier software engineering would be if there were a period where things are simply not available. What problems are currently very difficult would be made trivial if 6 hours of downtime every Sunday were acceptable? 10PM-4AM EST

I find it very hard to come up with a use case where a weekly 6 hours of downtime at night EST would be acceptable. US government websites already often do this, and I find that completely unacceptable since these services need to be available to anyone, regardless of work and life schedules. Any website that's international would also suffer greatly for an EST centralized scheduled downtime. Maybe a very localized w…

Why wouldn’t it be acceptable? People aren’t available all the time either, nor are stores. In fact software is unique in its availability.

Re: Zero downtime Postgres migration, done right

#27

I wonder how much easier software engineering would be if there were a period where things are simply not available. What problems are currently very difficult would be made trivial if 6 hours of downtime every Sunday were acceptable? 10PM-4AM EST

Your team is now working 10pm to 4am est every Sunday, and anyone who uses your services in any time zone around the world is without your service. 10pm est is lunch time in Australia. 4am est is working hours in many parts of western Europe.

You don't have to be huge to have customers all around the world.

Re: Zero downtime Postgres migration, done right

#28

Earlier quoted context omitted.

Then hopefully everything is HA in your company.

It absolutely is, BGP with geographically diverse paths, databases, app servers, etc, are all redundant. It's hosted in-house, so there is a cold standby database in AWS which would only be used if, say, an aircraft crashed into our server rooms.

Could you share what domain your company is in? Just curious what has such HA requirements

Re: Zero downtime Postgres migration, done right

#29

I wonder how much easier software engineering would be if there were a period where things are simply not available. What problems are currently very difficult would be made trivial if 6 hours of downtime every Sunday were acceptable? 10PM-4AM EST

Interestingly, this is unacceptable in the US and probably most of Western Europe. However, I know of more than one bank in Eastern Europe where online banking simply doesn't work after midnight, until about 6am.

Re: Zero downtime Postgres migration, done right

#30

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

pg_upgrade with --hard-link option will upgrade the db in mins, I've done a 2TB db in less than a minute. Yes, there is no rollback, but if you do enough testing prior it can be really smooth. Am in the middle of doing this across an environment where we have tens of hosts from a few GB to couple TB.
Post reply on HN