[dead]
We migrated our PostgreSQL database with 11 seconds downtime
91–100 of 210 posts
Re: We migrated our PostgreSQL database with 11 seconds downtime
#92Very clear and concise blog! Loved reading it. I'd be very curious to see how Amazon DMS performs at scale. Scale includes either of these situations:
a) Larger dataset - 2+TB
b) Higher throughputs - WAL Spikes (at least 2-3K TPS)
c) Reliably and efficiently replicating TOAST Columns (ex: large JSONBs).
d) Advance data-types - BLOB, CLOB, HSTORE, ARRAYs etc.
In my previous role with the Azure Postgres team, we observed that Azure DMS worked pretty seamlessly in migrations that did not involve the above situations. Once either of the above came in, the replication experience was hairy. Interestingly, in my current gig at PeerDB, many use-cases we are seeing have either of the above situations. We have been doing quite a surgical optimizations to handle those situations and they keep coming! That is the beauty of Postgres, enabling a myriad of workloads, each of the unique in their own way! :)
Re: We migrated our PostgreSQL database with 11 seconds downtime
#93Minor detail but did anyone else notice they are using eu-west-1? Kind of weird for the UK to be hosting there sites in another country (Ireland). I'm sure this isn't super sensitive data but still.
The London AWS region was missing a lot of features for a long time. It's easier to just go with a big region.
"Lets go with the easier approach" is a bit worrying when it means potentially breaking security assumptions / legal assurances.
Though I guess it could be explained away by having something in the website's "Terms and Conditions for use". ;)
Re: We migrated our PostgreSQL database with 11 seconds downtime
#94We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…
I also used RDS Blue/Green deployment to apply a MySQL major engine version upgrade from 5.7 to 8.0. With respect to downtime it worked fantastically, I think we measured 13 seconds of observable downtime from the API.
However we did learn the hard way that RDS Blue/Green cannot be used to apply arbitrary changes. In our case, we discovered RDS Blue/Green can only be used to move up engine versions, not down.
We discovered on MySQL 8.0 one of our stored procedures had very occasional failures, and considered the option of using RDS Blue/Green again to move back down to 5.7. Turns out that's not an option.
Re: We migrated our PostgreSQL database with 11 seconds downtime
#95We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…
[deleted]
Re: We migrated our PostgreSQL database with 11 seconds downtime
#96We tried to work with DMS to migrate mysql to postgres, and it was a nightmare. Support was useless, and would often just not get back to us without prodding. Then them giving us canned responses unrelated to our queries. The whole thing is nigh on un-debuggable. Stay away.
[deleted]
Re: We migrated our PostgreSQL database with 11 seconds downtime
#97Earlier quoted context omitted.
That sentence was a little confusing. You're not happy that the government is hiring experts to run an important service?
Yes. RDS is a very reasonable choice if you are a tech company, let alone a govt org. The alternative isn’t “let’s host this ourselves” it is “let’s host this with Oracle at a much higher cost”.
The alternative - at government scale - is absolutely 'let's host this ourselves' and that's what they should be doing, to ensure that institutional expertise remains. They should also own and operate their own datacentres which should be physically secure, not shared with commercial ventures and guarded by the armed forces, not civilian security.
Re: We migrated our PostgreSQL database with 11 seconds downtime
#98I'm quite negatively surprised that a government service is moving from their own platform to AWS for such an important service.
That sentence was a little confusing. You're not happy that the government is hiring experts to run an important service?
> is hiring experts
'moving to AWS' (or any cloud provider) is not 'hiring experts' it's just outsourcing the risk to an entity that you, in the event of a genuine crisis, have no leverage over beyond 'we're going to stop paying you (once we migrate away from you which will take ten years)'
Re: We migrated our PostgreSQL database with 11 seconds downtime
#99We didn't even use red/green deployment; just had a multi-AZ deployment and migrated from 11.x -> 15.2 with about 30 seconds of downtime.
No dramas really, didn't even need logical replication.
Re: We migrated our PostgreSQL database with 11 seconds downtime
#100We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…
Has anyone encrypted the storage on a previously unencrypted RDS using Blue/Green?
We did the exact thing not too long ago, but we couldn't do it using blue/green. We were able to minimize downtime quite a bit, but it was on the order of minutes, not seconds. I wrote a little bit about the process here. I spent a lot of time prepping and running the migration, so happy to share any details if it's helpful.
https://phizzle.space/dbadmin/aws/postgres/2023/12/30/rds-en...