71–80 of 210 posts
Minor detail but did anyone else notice they are using eu-west-1? Kind of weird for the UK to be hosting there sites in another country (Ireland). I'm sure this isn't super sensitive data but still.
Earlier quoted context omitted.
I don't think you have any idea just how much it costs to run infrastructure at the reliability levels provided by AWS, and just how much investment it would require to get the ball rolling on this. A lot of people have a very unrealistic picture of what government budgets are like.
> I don't think you have any idea just how much it costs to run infrastructure at the reliability levels provided by AWS my $12/year VPS does better than us-east-1
Note that the enemy of low/zero downtime migrations like this is long running queries. Ie. a single update query which takes 30 mins. You either have to kill and roll back that query, or suffer 30 mins of unavailability. As far as I know, there is no way to migrate a currently in progress query.
There are a lot of the former in the wild to be sure. I've had a lot of run turning minutes-hours into milliseconds. :)
Earlier quoted context omitted.
How would you feel if the US government ran on servers from a European company, which also works very hard to avoid paying taxes in US soil? All those reasons to go AWS hold for a private company, not for a government service of a first world country and G7 member. AWS has a lot of compliant services, but it's not like they're doing rocket science one of the top 5 richest countries in the world cannot afford to devel…
I would be surprised if they aren't deploying to the London data center, so I would think it is within the UK
* enabled our fallback "maintenance mode" site. It's a super lightweight version of our site running on CF workers.
* scaled down all apps using the db to 0 in terraform
* hit the upgrade button in the aws web ui, which runs pg_upgrade. 14->15
* waited for it to finish
* hit the upgrade button again. 15->16
* waited for the dbs to start accepting connections (they do before they're marked as ready, I think aws does more stuff than pg_upgrade)
* Started `VACUUM ANALYZE; REINDEX DATABASE CONCURRENTLY`. The idea is to avoid performance issues between versions and make use of performance improvements from new versions.
* Started spinning up the apps again
* Waited until all apps had a handful of containers running
* Started accepting traffic (disabled maintenance site)
* Went to bed
If I was doing this myself I wouldn't pay the AWS tax, instead do blue/green myself with logical replication and a load balancer.
We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…
+1 for B/G. That said, I imagine they couldn’t use it due to the cross-account shift. I’ve used it for both MySQL (with much higher QPS than TFA, by two orders of magnitude) and Postgres, and they both went flawlessly. Read the docs, especially the limitations. Re-read them. Do a test run in the dev environment, under load. Do it again in staging. Or just YOLO into prod ‘cause it’ll probably be fine, I guess.
We tried to work with DMS to migrate mysql to postgres, and it was a nightmare. Support was useless, and would often just not get back to us without prodding. Then them giving us canned responses unrelated to our queries. The whole thing is nigh on un-debuggable. Stay away.
We tried to work with DMS to migrate mysql to postgres, and it was a nightmare. Support was useless, and would often just not get back to us without prodding. Then them giving us canned responses unrelated to our queries. The whole thing is nigh on un-debuggable. Stay away.
Surprised by that. I've used AWS DMS quite a lot to do both on-prem to AWS and AWS (MySQL) to AWS Postgres migrations and long term ongoing replication. Whilst there is some complexity/gotchas there it's always been more than up to the task. Takes a little bit of validation/testing to understand but it's very well documented too. What sort of issues did you hit? In all honesty I'm not sure I've been more impressed by…
Earlier quoted context omitted.
The risk is hiring a team of ineffective sysadmins, especially if your organization can’t assess sysadmin competence.
That would indeed be a risk, but the circular logic of this means no new company could ever have any competence outside of its founders. Which feels shortsighted. Anyway, I am a former sysadmin. I am confident that I can identify competence in the requisite areas.
At a company if a department isn't working out you just restructure and move on, but in the government, that team is going to retire in your org and collect pension from you, and there's very little you can do about that.
We tried to work with DMS to migrate mysql to postgres, and it was a nightmare. Support was useless, and would often just not get back to us without prodding. Then them giving us canned responses unrelated to our queries. The whole thing is nigh on un-debuggable. Stay away.
I worked on migrating our MySQL system to PostgreSQL using pgloader ( https://pgloader.io/ ). There were some hiccups, things that needed clarification in documentation, and some additional processes that needed to be done outside of the system to get everything we need in place, it was a amazing help. Not sure the project would've been possible without it. Data mapping from PostgreSQL to PostgreSQL as in the article…
Would recommend pgloader.