There are various ways to 'pause' incoming postgres queries, for example using pgbouncer, - ie. don't fail them, simply delay them until the replication has caught up and then let them continue on the new database. If anything goes wrong and replication doesn't catch up, you can unpause and let those queries happen on the old database. Therefore, your 11 seconds of downtime becomes 0 to 11 seconds of added page load…
We migrated our PostgreSQL database with 11 seconds downtime
181–190 of 210 posts
Re: We migrated our PostgreSQL database with 11 seconds downtime
#182Earlier quoted context omitted.
Database changes are typically one-way. If your new change includes creating or modifying a table, such that there are new additional columns, and you populate those with data, then downgrading would destroy the changed columns and the data in them. Hence you can't downgrade once you upgrade or you'd potentially be breaking things. To downgrade safely you'd need to backup or snapshot the old database, and then restor…
DB schema migration script frameworks (at least in Python, Ruby & Java lands) do typically support both upgrade and downgrade directions. People skip implementing and testing the downgrade side if the development model doesn't need it but the problem of what happens to the data is controlled by what you put in the "down" migration script. I'd guess if you can't throw the data away, you won't do a down migration, you'…
For us this isn't a big deal though because we're writing our software so it should be able to function as expected on a DB with a newer schema. This makes upgrades much easier to handle has users can run new and old software side-by-side.
Re: We migrated our PostgreSQL database with 11 seconds downtime
#183I'm quite negatively surprised that a government service is moving from their own platform to AWS for such an important service.
And I'm not surprised if, they got some kickback, discount etc in some way to promote AWS on their blog. Not claiming its so, but I would not be surprised at all. It reads as one big advertisement.
Re: We migrated our PostgreSQL database with 11 seconds downtime
#184Earlier quoted context omitted.
DB schema migration script frameworks (at least in Python, Ruby & Java lands) do typically support both upgrade and downgrade directions. People skip implementing and testing the downgrade side if the development model doesn't need it but the problem of what happens to the data is controlled by what you put in the "down" migration script. I'd guess if you can't throw the data away, you won't do a down migration, you'…
> DB schema migration script frameworks (at least in Python, Ruby & Java lands) do typically support both upgrade and downgrade directions. They do, and in every shop I've ever been in these are considered a trap precisely because they don't consider data loss. Always roll forward. If you have to change migration history, restore a backup and lament past you's hubris.
A key component is that the schema management tool must be able to detect and warn/error on destructive changes -- regardless of whether it's a conceptual revert or just a bad change (i.e. altering a column's data type in a lossy way). My declarative tool Skeema [1] has handled this since the first release, among many other safety features.
That all said, schema changes are mostly orthogonal to database version upgrades, so this whole subthread is a bit different than the issue discussed several levels above :) The root of the blue/green no-rollback-after-upgrade issue discussed above is that MySQL logical replication officially supports older-version-primary -> newer-version-replica, but not vice versa. Across different release series, the replication format can change in ways that the older version replicas do not understand or support.
Re: We migrated our PostgreSQL database with 11 seconds downtime
#185We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…
Re: We migrated our PostgreSQL database with 11 seconds downtime
#186Earlier quoted context omitted.
Database changes are typically one-way. If your new change includes creating or modifying a table, such that there are new additional columns, and you populate those with data, then downgrading would destroy the changed columns and the data in them. Hence you can't downgrade once you upgrade or you'd potentially be breaking things. To downgrade safely you'd need to backup or snapshot the old database, and then restor…
DB schema migration script frameworks (at least in Python, Ruby & Java lands) do typically support both upgrade and downgrade directions. People skip implementing and testing the downgrade side if the development model doesn't need it but the problem of what happens to the data is controlled by what you put in the "down" migration script. I'd guess if you can't throw the data away, you won't do a down migration, you'…
Re: We migrated our PostgreSQL database with 11 seconds downtime
#187Lovely! We just migrated from postgres 14 to 16 for 3 postgres clusters (servers) on RDS containing about 2TB of data across 8 databases. We were down from 00:00 to 04:00. Steps we took: * enabled our fallback "maintenance mode" site. It's a super lightweight version of our site running on CF workers. * scaled down all apps using the db to 0 in terraform * hit the upgrade button in the aws web ui, which runs pg_upgra…
Have done 2 TB dbs in less than a minute.
We were running our own Postgres instances on-prem.
Re: We migrated our PostgreSQL database with 11 seconds downtime
#188We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…
Am waiting for them to support upgrades from RDS Postgres to Aurora.
Re: We migrated our PostgreSQL database with 11 seconds downtime
#189We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…
Trade off: For a few seconds some requests were slower.
DNS Groups w/ retries is a nifty mechanism for these things.
Tool used: https://github.com/shayonj/pg_easy_replicate
Re: We migrated our PostgreSQL database with 11 seconds downtime
#190Interesting, though I have no idea why the government is using AWS in the first place. This isn't a startup hacking away trying to find PMF, or dealing with unpredictable marketing-driven traffic spikes. We know we need these services running long term, and can make solid predictions about usage patterns. We could build a public sector cloud and/or adopt a sensible on-prem approach. This requires funding, coordinatio…
The government likely hired a consulting firm and AWS was part of the solution proposed and bought.