Status.github.com: “We're failing over a data storage system”

I am not the OP, but in my previous job, we ran infrastructure that could absolutely NOT be cloud hosted (network management control plane for an ISP), and had to be geo-redundant (3 DCs in our country, which also hosted the core routers and other data-plane components). The ISPs applications also ran on the same infrastructure.

So, we already had a 3-DC virtualisation platform.

We also chose Gitlab, with a primary in our primary DC (where dev/QA infrastructure resided), with daily full backups rsynced (as created by Gitlab) to the standby gitlab instance (and backups from the stadnby rsynced to the primary) and repos sync hourly. Plus, all volumes on the SAN-attached storage array were snapshotted and backed up to virtual tape in both sites.

The 3rd DC was too small for the same footprint, but had a smaller SAN array and we would be able to host the gitlab instance there in ~1 hour if both the primary and standby failed.

Total cost: * 2 VMs with 8GiB ram. All other costs were negligible. * 2x50GiB SAN volumes (at ~ $1/GB for capex+5-year-opex) * Some virtual tape costs (negligible to our other backup costs) * Time: * Approx 1 hour to set gitlab up * Approx 1 hour to setup backup replication and test restores * Approx 1 hour spent on Gitlab upgrades per year * Benefits: Integration with other on-premise infrastructure we would prefer not to expose to the 'cloud' via e.g. VPC gateways etc. * Unlimited (except by ram/storage assigned to VMs) private repos * Unlimited (except by ram/storage assigned to VMs) * Ability to deploy applications regardless of any other external outage (network, cloud etc.): priceless

I am not aware of any failures in the first 3 years of operation.

> What did the server cost? Negligible, we already had 12 physical hosts with 256GiB ram each in the primary and secondary site

> How many hours did you spend on it? 3 more hours than we would have if we had used a hosted solution, if the hosted solution would have cost 0 hours to integrate with our on-premise infrastructure in non-internet-exposed DMZs.

> What's your SLA?

Our standard SLA was 98.5% availability of all internal services measured monthly with 1h MTTR.

> Who will be woken up at night when there's an outage?

Same team that gets woken up for the other ~200 VMs that are more critical than the git repo

> What's your backup and recovery procedure?

Assuming we couldn't recover the primary: * Ensure the primary gitlab instance isn't taking writes/updates (e.g. disable the virtual NIC) * Start a restore from the lastest backup on the standby * Check if there were any commits to the git repos since the last backup, of so, push them to the recovered gitlab instance from the clones * Check that the repos are accessible and working * Flip the DNS from git.company to point to git.secondary-dc.company instead of git.primary-dc.company

We built everything to be able to failed over from the primary to secondary site and had tested all failovers (application servers, multiple databases, authentication servers, monitoring, infrastructure management applications etc.).

> but it's a bit naive to believe it's better than a hosted solution right away.

Depends on your use case and existing infrastructure/skills/operational posture etc.

Split brain can run for a while if the all identifiers are UUIDs and tables are used as append only. Restoration is complex, though.

They got to the end of the restore, realised there was a problem, and had to start over.

> But it's Monday morning in Australia and we need to release!

Ironic isn't it? The whole point of git was to be distributed, and yet we're at a place where a bunch of companies can't deploy software when a single git provider is down. I myself am in the same boat. Sure, I could reconfigure for a different repo, but is it even worth the effort?

I think it is obvious what happened. They must have moved their backend to Windows 10.