Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We migrated our PostgreSQL database with 11 seconds downtime

gds.blog.gov.uk

Post reply on HN
Print view View original

61–70 of 210 posts

Re: We migrated our PostgreSQL database with 11 seconds downtime

#61

Post by ComputerGuru »

> The second change was to create a DNS record in AWS Route53 for `database.notifications.service.gov.uk` with a 1 second TTL (time to live) [..] our migration script just needed to update the DNS weighting in AWS to 100% of results being sent to the target database location and wait 1 second for the TTL to expire. Then, when our apps next try to query our database they will be querying our target database.

Wait. Their (or Python's default?) db orm blocks while it performs a DNS lookup for each and every query!? It doesn't cache resolved addresses for any length of time? No connections are pooled and reused?

Re: We migrated our PostgreSQL database with 11 seconds downtime

#62

Post by blibble »

everfrustrated wrote: post #57
lnxg33k1 wrote: post #13

Earlier quoted context omitted.

Its not an organisation, its a blucking government, it handles citizen data, and its sending them to a company of foreign country, because it can’t hire some system administrators? A GOVERNMENT? What are they doing? Still looking for their product market fit and can’t afford the headcount? Is it a joke? EDIT If they are looking for money id like to participate a bit in the seed round

The UK replies entirely on the mercy of USA for its nuclear deterrent (Trident) For the UK at least, that ship has _long_ since sailed....

trident has UK built warheads and is operationally independent of the US

Re: We migrated our PostgreSQL database with 11 seconds downtime

#63

Post by DylanDmitri »

dijit wrote: post #24
lnxg33k1 wrote: post #13

Earlier quoted context omitted.

Its not an organisation, its a blucking government, it handles citizen data, and its sending them to a company of foreign country, because it can’t hire some system administrators? A GOVERNMENT? What are they doing? Still looking for their product market fit and can’t afford the headcount? Is it a joke? EDIT If they are looking for money id like to participate a bit in the seed round

Sysadmins are cheaper than many people seem to think. I had a person I trust a lot telling me that "if we go with a bare metal provider like GCore we'd have to hire someone", his reason for bringing that up was that the cost difference would be justified by not having to hire someone,. However a GCore €400,000k/y bill becomes a €6,000,000~ if you were to use a public cloud, even with the scaling up and down when not…

The risk is hiring a team of ineffective sysadmins, especially if your organization can’t assess sysadmin competence.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#66

Post by lozenge »

ClassicOrgin wrote: post #64

Minor detail but did anyone else notice they are using eu-west-1? Kind of weird for the UK to be hosting there sites in another country (Ireland). I'm sure this isn't super sensitive data but still.

The London AWS region was missing a lot of features for a long time. It's easier to just go with a big region.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#67

Post by overstay8930 »

Edwinr95 wrote: post #2

I'm quite negatively surprised that a government service is moving from their own platform to AWS for such an important service.

If you saw how non-tech companies run datacenters, well let's just say they're not exactly working with NATO like the big 3 cloud providers do when designing their DCs and backbone.

Honestly you should be frightened when you see someone NOT using a cloud provider, because it is hard work to properly run and secure a datacenter. Even Equinix fucks up HARD regularly and they are considered the gold standard (shout out to those I saw at 350 E Cermak over the weekend).

Re: We migrated our PostgreSQL database with 11 seconds downtime

#68

Post by dijit »

DylanDmitri wrote: post #63
dijit wrote: post #24

Earlier quoted context omitted.

Sysadmins are cheaper than many people seem to think. I had a person I trust a lot telling me that "if we go with a bare metal provider like GCore we'd have to hire someone", his reason for bringing that up was that the cost difference would be justified by not having to hire someone,. However a GCore €400,000k/y bill becomes a €6,000,000~ if you were to use a public cloud, even with the scaling up and down when not…

The risk is hiring a team of ineffective sysadmins, especially if your organization can’t assess sysadmin competence.

That would indeed be a risk, but the circular logic of this means no new company could ever have any competence outside of its founders. Which feels shortsighted.

Anyway, I am a former sysadmin. I am confident that I can identify competence in the requisite areas.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#69

Post by extesy »

Topgamer7 wrote: post #42

We tried to work with DMS to migrate mysql to postgres, and it was a nightmare. Support was useless, and would often just not get back to us without prodding. Then them giving us canned responses unrelated to our queries. The whole thing is nigh on un-debuggable. Stay away.

I can confirm that. Both DMS reliability and support quality were terrible.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#70

Post by sgarland »

Ozzie_osman wrote: post #58

We did a similar migration (somewhat larger database) with ~20 seconds of downtime and much less work... using the magic of AWS RDS Blue-Green Deployments [1]. Surprised they aren't mentioned in the thread yet. Basically, you spin up a new Blue Green deployment with any desired changes (in our case, we were upgrading Postgres major from 13 to 15). While your blue configuration continues to serve traffic, AWS uses log…

+1 for B/G. That said, I imagine they couldn’t use it due to the cross-account shift. I’ve used it for both MySQL (with much higher QPS than TFA, by two orders of magnitude) and Postgres, and they both went flawlessly.

Read the docs, especially the limitations. Re-read them. Do a test run in the dev environment, under load. Do it again in staging.

Or just YOLO into prod ‘cause it’ll probably be fine, I guess.

Post reply on HN