Thank you for redacting who the engineer was. Great write-up. Thank you!
Postmortem of database outage of January 31
11–20 of 269 posts
Re: Postmortem of database outage of January 31
#12>An ideal environment is one in which you can make mistakes but easily and quickly recover from them with minimal to no impact.
This is a great attitude. Too often opportunity cost isn't considered when making rules to protect folks from doing something stupid.
Re: Postmortem of database outage of January 31
#13I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second.
Also, RDS gives you a synchronously replicated standby database, and automates failover, including updating the DNS CNAME that the clients connect to during a failover (so it is seamless to the clients, other than requiring a reconnect), and ensuring that you don't lose a single transaction during a failover (the magic of synchronous replication over a low latency link between datacenters).
For a company like Gitlab, that is public about wanting to exit the cloud, I feel like they could have really benefited from a fully managed relational database service. This entire tragic situation could have never happened if they were willing to acknowledge the obvious: managing relational databases is hard, and allowed someone with better operational automation, like AWS, to do it for them.
Re: Postmortem of database outage of January 31
#14It's amazing how quickly it descends into "one of the engineers" did x or y. Who was steering this ship exactly?
It's really simple to point the finger and try to find a single cause of failure - but it's a fools errand - comparable to finding the single source behind a great success.
Re: Postmortem of database outage of January 31
#15Thanks so much for the post and transparency Gitlab! We had just finished recovering from our own outage (stemming for a power loss and subsequent cascading failures) and were scheduled to do our post-mortem on 2/1 so the original document was a refreshing and reassuring read.
Re: Postmortem of database outage of January 31
#16Earlier quoted context omitted.
We too are glad we had those snapshots. And while it was the worst thing that ever happend at GitLab it is humbling to know that it could have been worse.
What do you think would have happened if you had total data loss/failure?
I don't work for gitlab.
Isn't that a put you out of business moment? I guess there is enterprise installations but the main website would be toast.
I guess there is the chance there were still older backups taken intentionally or by chance that would have been worse but recoverable from
Re: Postmortem of database outage of January 31
#17I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…
A great number of issues can be attributed to the selection of Azure as the platform of choice. That said, a little bird told me that the decision was largely a cost factor. "You get what you pay for" never rang more true.
Re: Postmortem of database outage of January 31
#18>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from readin…
Misread that as "The engineer was terminated" at first. Poor guy.
Re: Postmortem of database outage of January 31
#19It's amazing how quickly it descends into "one of the engineers" did x or y. Who was steering this ship exactly? It's really simple to point the finger and try to find a single cause of failure - but it's a fools errand - comparable to finding the single source behind a great success.
Do you expect management to be staring over your shoulder every time you do some kind of `rm` on a production server? With great power comes great responsibility.
Re: Postmortem of database outage of January 31
#20>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from readin…
For me, it was when I meant to execute this...
rm -rf ~/foo
rm -rf ~ /foo