Postmortem of database outage of January 31
about.gitlab.com
Postmortem of database outage of January 31
1–10 of 269 posts
Re: Postmortem of database outage of January 31
#2I found this entertaining, even if they did later admit that it was a hoax:
http://serverfault.com/questions/587102/monday-morning-mista...
Re: Postmortem of database outage of January 31
#3Reading this, the thing that stuck out to me was how remarkably lucky they were to have the two snapshots. The one from 6 hours earlier was there seemingly by chance, as an engineer had created it for unrelated reasons. And for both the 6- and 24-hour snapshots, it seems just lucky that neither had any breaking changes made to them by pre-production code (they _were_ dev/staging snapshots, after all).
I'm glad it all worked out in the end!
Re: Postmortem of database outage of January 31
#4Reading this, the thing that stuck out to me was how remarkably lucky they were to have the two snapshots. The one from 6 hours earlier was there seemingly by chance, as an engineer had created it for unrelated reasons. And for both the 6- and 24-hour snapshots, it seems just lucky that neither had any breaking changes made to them by pre-production code (they _were_ dev/staging snapshots, after all). I'm glad it all…
We too are glad we had those snapshots. And while it was the worst thing that ever happend at GitLab it is humbling to know that it could have been worse.
Re: Postmortem of database outage of January 31
#5Am I missing something or didn't they mention 'test recovery, not backups'?
Re: Postmortem of database outage of January 31
#6Reading this, the thing that stuck out to me was how remarkably lucky they were to have the two snapshots. The one from 6 hours earlier was there seemingly by chance, as an engineer had created it for unrelated reasons. And for both the 6- and 24-hour snapshots, it seems just lucky that neither had any breaking changes made to them by pre-production code (they _were_ dev/staging snapshots, after all). I'm glad it all…
We too are glad we had those snapshots. And while it was the worst thing that ever happend at GitLab it is humbling to know that it could have been worse.
What do you think would have happened if you had total data loss/failure?
Re: Postmortem of database outage of January 31
#7GitHub also lost a bunch of PRs and issues sitewide early in their history. They claimed to have restored all the PRs from backup, but I was pretty sure I had opened a PR and it never came back. I emailed support and they basically told me tough luck.
Re: Postmortem of database outage of January 31
#8yes, wal archiving would have helped (archive_command = rsync standby ...), but it's also very easy in postgres 9.4+ to add a replication slot on the master so that wal is kept until it is no longer needed by the standby. simply reference the slot in the standby's recovery.conf file.
definitely monitor your replication lag--or at least disk usage on the master--with this approach (in case wal starts piling up there).
Re: Postmortem of database outage of January 31
#9sytse and GitLab folks: thank you for the transparency.
Re: Postmortem of database outage of January 31
#10>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed.
I could feel the sweat drops just from reading this.
I'd bet every one of us has experienced the panicked Ctrl+C of Death at some point or another.