Live data from Hacker News

We migrated our PostgreSQL database with 11 seconds downtime

gds.blog.gov.uk

181–190 of 210 posts

Re: We migrated our PostgreSQL database with 11 seconds downtime

#181

There are various ways to 'pause' incoming postgres queries, for example using pgbouncer, - ie. don't fail them, simply delay them until the replication has caught up and then let them continue on the new database. If anything goes wrong and replication doesn't catch up, you can unpause and let those queries happen on the old database. Therefore, your 11 seconds of downtime becomes 0 to 11 seconds of added page load…

It’s one thing to pause queries but can you also pause transactions that are in flight? How does that work?

Re: We migrated our PostgreSQL database with 11 seconds downtime

#182

Earlier quoted context omitted.

Database changes are typically one-way. If your new change includes creating or modifying a table, such that there are new additional columns, and you populate those with data, then downgrading would destroy the changed columns and the data in them. Hence you can't downgrade once you upgrade or you'd potentially be breaking things. To downgrade safely you'd need to backup or snapshot the old database, and then restor…

DB schema migration script frameworks (at least in Python, Ruby & Java lands) do typically support both upgrade and downgrade directions. People skip implementing and testing the downgrade side if the development model doesn't need it but the problem of what happens to the data is controlled by what you put in the "down" migration script. I'd guess if you can't throw the data away, you won't do a down migration, you'…

Our in-house schema migration tool supports downgrading, but it won't remove non-empty tables or columns etc.

For us this isn't a big deal though because we're writing our software so it should be able to function as expected on a DB with a newer schema. This makes upgrades much easier to handle has users can run new and old software side-by-side.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#183
post #2

I'm quite negatively surprised that a government service is moving from their own platform to AWS for such an important service.

"The PaaS team offered us the ability to migrate databases using AWS Database Migration Service (DMS)."

And I'm not surprised if, they got some kickback, discount etc in some way to promote AWS on their blog. Not claiming its so, but I would not be surprised at all. It reads as one big advertisement.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#184

Earlier quoted context omitted.

DB schema migration script frameworks (at least in Python, Ruby & Java lands) do typically support both upgrade and downgrade directions. People skip implementing and testing the downgrade side if the development model doesn't need it but the problem of what happens to the data is controlled by what you put in the "down" migration script. I'd guess if you can't throw the data away, you won't do a down migration, you'…

> DB schema migration script frameworks (at least in Python, Ruby & Java lands) do typically support both upgrade and downgrade directions. They do, and in every shop I've ever been in these are considered a trap precisely because they don't consider data loss. Always roll forward. If you have to change migration history, restore a backup and lament past you's hubris.

This is solved more cleanly in declarative schema management systems, where you have a schema repo of CREATE statements, and the tool can auto-generate the correct DDL. You never need to write any migrations at all, up or down. If you need to roll back, you use `git revert` and then auto-generate from there. The history is in Git, and you can fully leverage Git like a proper codebase.

A key component is that the schema management tool must be able to detect and warn/error on destructive changes -- regardless of whether it's a conceptual revert or just a bad change (i.e. altering a column's data type in a lossy way). My declarative tool Skeema [1] has handled this since the first release, among many other safety features.

That all said, schema changes are mostly orthogonal to database version upgrades, so this whole subthread is a bit different than the issue discussed several levels above :) The root of the blue/green no-rollback-after-upgrade issue discussed above is that MySQL logical replication officially supports older-version-primary -> newer-version-replica, but not vice versa. Across different release series, the replication format can change in ways that the older version replicas do not understand or support.

[1] https://github.com/skeema/skeema

Re: We migrated our PostgreSQL database with 11 seconds downtime

#185

We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…

I used this about 2 months ago going from MySql 5.7->8.0 Really awesome feature.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#186

Earlier quoted context omitted.

Database changes are typically one-way. If your new change includes creating or modifying a table, such that there are new additional columns, and you populate those with data, then downgrading would destroy the changed columns and the data in them. Hence you can't downgrade once you upgrade or you'd potentially be breaking things. To downgrade safely you'd need to backup or snapshot the old database, and then restor…

DB schema migration script frameworks (at least in Python, Ruby & Java lands) do typically support both upgrade and downgrade directions. People skip implementing and testing the downgrade side if the development model doesn't need it but the problem of what happens to the data is controlled by what you put in the "down" migration script. I'd guess if you can't throw the data away, you won't do a down migration, you'…

[deleted]

Re: We migrated our PostgreSQL database with 11 seconds downtime

#187
post #75

Lovely! We just migrated from postgres 14 to 16 for 3 postgres clusters (servers) on RDS containing about 2TB of data across 8 databases. We were down from 00:00 to 04:00. Steps we took: * enabled our fallback "maintenance mode" site. It's a super lightweight version of our site running on CF workers. * scaled down all apps using the db to 0 in terraform * hit the upgrade button in the aws web ui, which runs pg_upgra…

I would just use pg_upgrade with --hardlinks for an in-place upgrade.

Have done 2 TB dbs in less than a minute.

We were running our own Postgres instances on-prem.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#188

We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…

Blue/green is new, so am guessing most folks don't know about it. It's the way for close to zero downtime upgrades.

Am waiting for them to support upgrades from RDS Postgres to Aurora.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#189

We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…

+1 for Route53 Groups and B/G setups. We did something similar with PG upgrades, no downtime with AWS R53 groups & retry inflight queries with a custom Rails ActiveRecord transaction patch.

Trade off: For a few seconds some requests were slower.

DNS Groups w/ retries is a nifty mechanism for these things.

Tool used: https://github.com/shayonj/pg_easy_replicate

Re: We migrated our PostgreSQL database with 11 seconds downtime

#190

Interesting, though I have no idea why the government is using AWS in the first place. This isn't a startup hacking away trying to find PMF, or dealing with unpredictable marketing-driven traffic spikes. We know we need these services running long term, and can make solid predictions about usage patterns. We could build a public sector cloud and/or adopt a sensible on-prem approach. This requires funding, coordinatio…

The government likely hired a consulting firm and AWS was part of the solution proposed and bought.

This doesn’t sound like how GDS operates.
Post reply on HN