Zero downtime Postgres migration, done right • Hacker News Board

Re: Zero downtime Postgres migration, done right

#131

Post by rigaspapas » Tue, Jun 15, 2021, 10:08 AM UTC

ivan888 wrote: post #44

I didn't read this article, but I really hate the tag line "done right". It expresses such a poor sense of humility, which is one of, or perhaps the most, important traits in the world of software

We didn't mean to be arrogant with the "done right" statement. In the background story we explain how we performed the same migration once again in the past, and we end up with data loss. So this was the time that actually "did it right". Many tutorials on the Internet that describe a similar process have flows that also lead to data loss.

Re: Zero downtime Postgres migration, done right

#132

Post by rigaspapas » Tue, Jun 15, 2021, 10:14 AM UTC

jeffbee wrote: post #63

Bucardo has no performance impact, it just adds a trigger to every mutation. Negligible! I really think articles of this kind are not useless, but need to explicitly narrow their audience to set expectation at the start. This particular topic is PG replication for users who aren't very sensitive to write latency.

That's a good point. We mention latency as "replication lag" and we have devoted a paragraph to the drift that comes as the result of this latency. In our case, we measured the latency to be <1s, and it was totally acceptable. As for the performance, triggers can become a problem with enough write traffic. In a different use case, Bucardo would be eliminated from a potential solution due to this.

Re: Zero downtime Postgres migration, done right

#133

Post by rigaspapas » Tue, Jun 15, 2021, 10:18 AM UTC

jlmorton wrote: post #20

If you are in AWS, or have connectivity available, AWS Database Migration Service makes this relatively trivial. DMS for Postgres is based on Postgres Logical Replication, which is built-in to Postgres, and the same thing Bucardo is using behind the scenes. But AWS DMS is very nearly point-and-click to do this sort of migration.

You are right. AWS DMS was our very first choice to try out. It is very easy to use, deployed within your VPC and most of the problems we mention in the article are already solved. Unfortunately, we experienced errors during our tests and the logging mechanisms were not quite helpful, so we failed to find out the problem and make it work.

Re: Zero downtime Postgres migration, done right

#134

Post by rigaspapas » Tue, Jun 15, 2021, 11:14 AM UTC

silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

A few minutes of downtime wouldn't be that bad, but it would harm the company's prestige. Unlike AWS and Netflix, we are not based on massive consumption, but on a few customers. Losing one good customer can make a difference. We believe it worth the investment of time.

Re: Zero downtime Postgres migration, done right

#135

Post by rigaspapas » Tue, Jun 15, 2021, 11:15 AM UTC

silviogutierrez wrote: post #7

Very interesting article. But I have to ask: would taking down the system for a couple of hours be that bad? I looked at the company, and while they seem rather large, they're not Netflix or AWS. I imagine they need to be up for people to be able to check in, etc. But they could just block out the planned maintenance as check in times far in advance. I'm sure there's a million other edge cases but those can be though…

Update: The cost is a good point. It wasn't that high. We actually updated the article to reflect this. It was two engineers, working full-time on it, for three weeks.