Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We migrated our PostgreSQL database with 11 seconds downtime

gds.blog.gov.uk

Post reply on HN
Print view View original

31–40 of 210 posts

Re: We migrated our PostgreSQL database with 11 seconds downtime

#31

Post by foofie »

lnxg33k1 wrote: post #13
conception wrote: post #9

Earlier quoted context omitted.

AWS has a lot of pre-audited compliance built into their services. Being able to inherit their certification for services can save an organization a lot of time and effort.

Its not an organisation, its a blucking government, it handles citizen data, and its sending them to a company of foreign country, because it can’t hire some system administrators? A GOVERNMENT? What are they doing? Still looking for their product market fit and can’t afford the headcount? Is it a joke? EDIT If they are looking for money id like to participate a bit in the seed round

> (...) its a blucking government, it handles citizen data, and its sending them to a company of foreign country, because it can’t hire some system administrators? A GOVERNMENT? What are they doing?

This is a very good question, and bears repeating.

It's not a massive database as well. 400GB with 1k inserts/second.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#32

Post by robertlagrant »

swozey wrote: post #21
jazzyjackson wrote: post #4

Earlier quoted context omitted.

I don't know what it's like in UK but it may be the case that government has a hard time a{ttract,fford}ing talent to administer everything in house. Not that AWS is great for cost saving but if its between paying 50k/year for cloud services and not being able to find an engineer who will competently do the job for less than 50k, then the cloud is your only move really.

They require various clearances (digging into your life and past relationships to a miserable degree), don't allow someone to have ever smoked pot and pay half or less of what you can make in the pvt sector here (usa). Everyone I know working FedRAMP jobs is prior military/g-level.

They wouldn't need that. And having been SC cleared in the UK, and known a few DV-cleared ones, at least in the UK they don't care if you've smoked pot. They just care that if you have, that you don't mind your family knowing one day. They don't want people who can be blackmailed.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#33

Post by foofie »

pixelesque wrote: post #19
Edwinr95 wrote: post #2

I'm quite negatively surprised that a government service is moving from their own platform to AWS for such an important service.

As other comments point out, their own platform was (at least in terms of DB) already running on AWS, just using a different account.

> As other comments point out, their own platform was (at least in terms of DB) already running on AWS, just using a different account.

That changes nothing. It just means this unjustifiable nonsense is going on for a while.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#34

Post by justsomehnguy »

jazzyjackson wrote: post #4
Edwinr95 wrote: post #2

I'm quite negatively surprised that a government service is moving from their own platform to AWS for such an important service.

I don't know what it's like in UK but it may be the case that government has a hard time a{ttract,fford}ing talent to administer everything in house. Not that AWS is great for cost saving but if its between paying 50k/year for cloud services and not being able to find an engineer who will competently do the job for less than 50k, then the cloud is your only move really.

> who will competently do the job for less than 50k, then the cloud is your only move really

Well, there is the other way, but, as we know, never ever that would happen.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#35

Post by [deleted] »

swozey wrote: post #21
jazzyjackson wrote: post #4

Earlier quoted context omitted.

I don't know what it's like in UK but it may be the case that government has a hard time a{ttract,fford}ing talent to administer everything in house. Not that AWS is great for cost saving but if its between paying 50k/year for cloud services and not being able to find an engineer who will competently do the job for less than 50k, then the cloud is your only move really.

They require various clearances (digging into your life and past relationships to a miserable degree), don't allow someone to have ever smoked pot and pay half or less of what you can make in the pvt sector here (usa). Everyone I know working FedRAMP jobs is prior military/g-level.

[deleted]

Re: We migrated our PostgreSQL database with 11 seconds downtime

#36

Post by robertlagrant »

bboygravity wrote: post #28
kunwon1 wrote: post #16

Earlier quoted context omitted.

AWS has a G-Cloud for UK just like they have one for US, no?

Why can't the UK government build there own cloud? It's just completely insane to me that they would make the gov internet infrastructure completely (geopolitically) dependent on another country AND just literally give all their (citizens') data away AND pay for that "privilege"?! I mean if the government can't host the government's websites using tech from the government's country, maybe it would be better to just f…

They'd still be outsourcing to a firm to do this. They wouldn't hire a load of people to do it in-house. See also Fujitsu in the recently-popular Horizon scandal, or the NHS for IT debacle[0].

[0] https://en.wikipedia.org/wiki/NHS_Connecting_for_Health

Re: We migrated our PostgreSQL database with 11 seconds downtime

#37

Post by postpawl »

mbb70 wrote: post #12

Interesting to compare this to https://knock.app/blog/zero-downtime-postgres-upgrades discussed here https://news.ycombinator.com/item?id=38616181 A lot of the discussion boiled down to 'this is a lot of complexity to avoid a few minutes of downtime'. I guess this is the proof, just use AWS Data Migration Service, swap the DNS entries to go live and live with 11 seconds of downtime.

There are a lot of gotchas with using DMS (which seems to use pglogical under the hood). Since it’s not hardware-level replication, you can run into issues with large rows/columns/tables and it doesn’t really handle foreign keys. It may not handle some special data types at all. You also need to update the sequences after the migration or you’ll get errors about duplicate primary keys. You can also have issues if you don’t have proper primary keys, because it doesn’t always copy the entire row at once.

If the databases are within the same AWS account, it’s likely easier to use hardware-level replication with global database or snapshots to do migrations if you’re ok with 4-5 mins of downtime.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#38

Post by londons_explore »

There are various ways to 'pause' incoming postgres queries, for example using pgbouncer, - ie. don't fail them, simply delay them until the replication has caught up and then let them continue on the new database.

If anything goes wrong and replication doesn't catch up, you can unpause and let those queries happen on the old database.

Therefore, your 11 seconds of downtime becomes 0 to 11 seconds of added page load time. But more importantly, of the thousands of users of the database who have never seen a query fail before and might have buggy error handling codepaths or have a single failed query ruin a whole batch job, this approach leads to a lot less collateral damage.

Re: We migrated our PostgreSQL database with 11 seconds downtime

#39

Post by vdaea »

bboygravity wrote: post #28
kunwon1 wrote: post #16

Earlier quoted context omitted.

AWS has a G-Cloud for UK just like they have one for US, no?

Why can't the UK government build there own cloud? It's just completely insane to me that they would make the gov internet infrastructure completely (geopolitically) dependent on another country AND just literally give all their (citizens') data away AND pay for that "privilege"?! I mean if the government can't host the government's websites using tech from the government's country, maybe it would be better to just f…

Why should they build their own cloud, seeing that costs more money?

Re: We migrated our PostgreSQL database with 11 seconds downtime

#40

Post by londons_explore »

Note that the enemy of low/zero downtime migrations like this is long running queries.

Ie. a single update query which takes 30 mins.

You either have to kill and roll back that query, or suffer 30 mins of unavailability.

As far as I know, there is no way to migrate a currently in progress query.

Post reply on HN