Live data from Hacker News

We migrated our PostgreSQL database with 11 seconds downtime

gds.blog.gov.uk

51–60 of 210 posts

Re: We migrated our PostgreSQL database with 11 seconds downtime

#51

We tried to work with DMS to migrate mysql to postgres, and it was a nightmare. Support was useless, and would often just not get back to us without prodding. Then them giving us canned responses unrelated to our queries. The whole thing is nigh on un-debuggable. Stay away.

This was my experience as well. We thought it was a nice managed way to move pg to pg with minimal setup but we ran into so many issues we just did logical replication with bespoke fixes for things it didn't handle well.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#52
post #45

Earlier quoted context omitted.

Why can't the UK government build there own cloud? It's just completely insane to me that they would make the gov internet infrastructure completely (geopolitically) dependent on another country AND just literally give all their (citizens') data away AND pay for that "privilege"?! I mean if the government can't host the government's websites using tech from the government's country, maybe it would be better to just f…

I don't think you have any idea just how much it costs to run infrastructure at the reliability levels provided by AWS, and just how much investment it would require to get the ball rolling on this. A lot of people have a very unrealistic picture of what government budgets are like.

> I don't think you have any idea just how much it costs to run infrastructure at the reliability levels provided by AWS

my $12/year VPS does better than us-east-1

Re: We migrated our PostgreSQL database with 11 seconds downtime

#55

We tried to work with DMS to migrate mysql to postgres, and it was a nightmare. Support was useless, and would often just not get back to us without prodding. Then them giving us canned responses unrelated to our queries. The whole thing is nigh on un-debuggable. Stay away.

Surprised by that. I've used AWS DMS quite a lot to do both on-prem to AWS and AWS (MySQL) to AWS Postgres migrations and long term ongoing replication. Whilst there is some complexity/gotchas there it's always been more than up to the task. Takes a little bit of validation/testing to understand but it's very well documented too.

What sort of issues did you hit? In all honesty I'm not sure I've been more impressed by another AWS service.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#57

Earlier quoted context omitted.

AWS has a lot of pre-audited compliance built into their services. Being able to inherit their certification for services can save an organization a lot of time and effort.

Its not an organisation, its a blucking government, it handles citizen data, and its sending them to a company of foreign country, because it can’t hire some system administrators? A GOVERNMENT? What are they doing? Still looking for their product market fit and can’t afford the headcount? Is it a joke? EDIT If they are looking for money id like to participate a bit in the seed round

The UK replies entirely on the mercy of USA for its nuclear deterrent (Trident)

For the UK at least, that ship has _long_ since sailed....

Re: We migrated our PostgreSQL database with 11 seconds downtime

#58
We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet.

Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses logical replication to keep the "green" deployment in-sync. You can keep modifying (or testing) the "green" deployment (eg you could load test it if you wanted to), as long as you don't do any writes to it (writes still have to go to your live, blue configuration, and are replicated to green).

When you're ready, you run the "switch" command, and AWS does a few things for you: run checks to ensure blue/green are in sync, stops writes and connections, waits a few seconds to ensure replication is caught up, renames your database, then allows connections/writes again. We had less than 20 seconds of downtime, by our count. And, we had a primary and several read replicas and AWS successfully switched the full configuration over with no hiccups. You don't even need to switch your configuration because AWS swaps the database URLs for you. Green becomes blue, blue becomes old blue, and when you're ready, you delete "old blue".

Highly recommend! They do have some restrictions (for instance, not sure if it would work if you're switching accounts, etc).

1. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/blue-...

Re: We migrated our PostgreSQL database with 11 seconds downtime

#59
post #54

I love that where the rubber meets the road it doesn’t matter if you’re a government organisation or a web agency, it’s still just plain old Postgres, python, sqlalchemy and dns with a low ttl.

To be fair, there's plenty of government orgs at the bottom and web agencies at the top.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#60
post #2

I'm quite negatively surprised that a government service is moving from their own platform to AWS for such an important service.

Why?

I've worked on a number of UK government projects, including some with particularly sensitive security and data requirements.

Having some knowledge of their on-prem data centres and UK Cloud offering they have also used moving to AWS has so many operational, security and resilience benefits that aren't available elsewhere. It's not a free-lunch by any means and needs thought and governance certainly but the procurement simplification benefits alone make going to the public cloud a no brainer for a lot of government services.

It is worth knowing that even the on-prem data centres are usually operated by 3rd parties such as HP, BT and IBM. There was an initiative to have "Crown-managed" data-centers but it's not particularly scalable.

Post reply on HN