Live data from Hacker News

Zero downtime Postgres migration, done right

engineering.theblueground.com

81–90 of 135 posts

Re: Zero downtime Postgres migration, done right

#81

Earlier quoted context omitted.

The existence of planned downtime doesn’t mean they have to work during said time. I’m not following your reasoning

they might not have to work during every planned downtime, but they're going to have to work during some of them, or what was the point of the downtime?

The point is to create a predictable period in which downtime, if any, would occur during

Re: Zero downtime Postgres migration, done right

#82
post #43

If you can afford a one off 1 second of latency for your SQL queries, then using logical replication with pgbouncer seems way easier : - setup logical replication between the old and the new server (limitations exist on what is replicated, read the docs) - PAUSE the pgbouncer (virtual) database. Your app will hang, but not disconnect from pgbouncer - Copy the sequences from the old to new server. Sequences are not re…

That's how you do it.

Re: Zero downtime Postgres migration, done right

#83

Earlier quoted context omitted.

they might not have to work during every planned downtime, but they're going to have to work during some of them, or what was the point of the downtime?

The point is to create a predictable period in which downtime, if any, would occur during

This predictability only matters for frequent, repeat customers. But those are the exact customers that you could just email saying "the website is going to be down on Sunday for 1 hour for planned maintenance" a few times a year when it's actually required.

If you run a business (eg. HFT where you run certain cleanup scripts after trading hours) where regular planned downtime is fine then go for it but most of the time, you don't need a downtime schedule every single week because you're not doing planned maintenance work every single week.

Re: Zero downtime Postgres migration, done right

#84
post #40

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

I think it's always worth questioning both sides. Why is downtime acceptable? People on this site routinely complain about windows needing a restart for system updates while boasting about their Linux servers uptime. People talk about how kubernetes is overkill for many people, but it gives you rolling deployments for your applications out of the box. There's also the "slippery slope" argument. A 0 downtime migration…

I think the slippery slope argument is almost always a fallacy. In practice you can say "we accept 99.99% availability" (which is about 1 hour every year) and budget for that. Your service might go down for unplanned reasons such as an outage but it could also go down for planned maintenance and as long as you're within that budget it seems ok.

Re: Zero downtime Postgres migration, done right

#85
post #71

Its insane that it has to be this complex and require third party software to accomplish… Most modern rdbms/nosql database vendors allow a rolling upgrade where you roll in new servers and roll out the old ones seamlessly. Also the fact that AWS rds doesnt do this with zero downtime by default through automating it this way is also crazy. Why pay for hosted when the upgrade story is incomplete? Take downtime to upgra…

Well, Postgres is F/OSS, so I expect the solution to problems to be "lots of small tools"... but I see the same kind of herculean battle-plans for MS SQL Server work, and I get shocked. That is a paid product, what on Earth are we paying for?

You're paying for an "if something goes wrong and it's the software's fault you can sue us" license

Re: Zero downtime Postgres migration, done right

#86
post #83

Earlier quoted context omitted.

The point is to create a predictable period in which downtime, if any, would occur during

This predictability only matters for frequent, repeat customers. But those are the exact customers that you could just email saying "the website is going to be down on Sunday for 1 hour for planned maintenance" a few times a year when it's actually required. If you run a business (eg. HFT where you run certain cleanup scripts after trading hours) where regular planned downtime is fine then go for it but most of the t…

I think people are getting too caught up in my particular example. It could be a month, biweekly, etc.

Re: Zero downtime Postgres migration, done right

#87

Its insane that it has to be this complex and require third party software to accomplish… Most modern rdbms/nosql database vendors allow a rolling upgrade where you roll in new servers and roll out the old ones seamlessly. Also the fact that AWS rds doesnt do this with zero downtime by default through automating it this way is also crazy. Why pay for hosted when the upgrade story is incomplete? Take downtime to upgra…

> Most modern rdbms/nosql database vendors

"Modern" = "not a traditional RDBMS". Expecting your 1983 Toyota Corolla to fly, regardless of the fact that it's now 2021, is unrealistic. But personal helicopters have been around for a while.

Re: Zero downtime Postgres migration, done right

#88
post #43

If you can afford a one off 1 second of latency for your SQL queries, then using logical replication with pgbouncer seems way easier : - setup logical replication between the old and the new server (limitations exist on what is replicated, read the docs) - PAUSE the pgbouncer (virtual) database. Your app will hang, but not disconnect from pgbouncer - Copy the sequences from the old to new server. Sequences are not re…

Exactly this. The OP’s approach reminded me so much of the days of Slony, and I wondered why a simpler approach with logical replication would not just suffice.

Rather than pgbouncer, I did this in the actual application code once (write to both databases at the same time, once everything is in sync and you’re confident the new server works well, fail over to the new one only), but it depends upon how much control you can exercise over the application code.

Any approach that is based on triggers makes me shiver, however.

Re: Zero downtime Postgres migration, done right

#89
post #70
post #66

Earlier quoted context omitted.

I think that's covered in this talk (I've not watched the video though): https://www.braintreepayments.com/blog/ruby-conf-australia-h...

It was also covered in: https://www.braintreepayments.com/blog/switching-datacenters...

Aha, that's the post I was looking for!

Re: Zero downtime Postgres migration, done right

#90
post #71

Earlier quoted context omitted.

Well, Postgres is F/OSS, so I expect the solution to problems to be "lots of small tools"... but I see the same kind of herculean battle-plans for MS SQL Server work, and I get shocked. That is a paid product, what on Earth are we paying for?

You're paying for an "if something goes wrong and it's the software's fault you can sue us" license

You're not. Approximately 25% of SQL Server EULA text deals with ways in which the warranty is limited. The best the license gives you is your money back if you prove that it was the software's fault.

Of course, you can always sue. A $2T company. Good luck with that.

Post reply on HN