While this is sad to see it is a lesson to us all and I've shared it with coworkers who haven't taken disaster recovery as seriously as we should on our projects. Hopefully this will help raise this as a priority.
GitLab Database Incident – Live Report
121–130 of 621 posts
Re: GitLab Database Incident – Live Report
#122Re: GitLab Database Incident – Live Report
#123Earlier quoted context omitted.
Also, as a safety net, sometimes you don't need to run `rm -rf` (a command which should always be prefaced with 5 minutes of contemplation on a production system). In this case, `rmdir` would have been much safer, as it errors on non-empty directories.
Or use `mv x x.bak` when `rmdir` fails
Or even instead of any kind of rm command. mv is less subtle. I tend to prefer
`mv x $(date +%Y%m%d_%s_)x`
%Y - 4 digit year
%m - 2 digit month
%d - 2 digit day
_ - underscore literal
%s - linux timestamp (seconds since epoch)
In case it's not apparent to some, this command moves the directory (or file) called
'x'
'20170131_1485916040_x'
Re: GitLab Database Incident – Live Report
#124Seems like very basic mistakes were made, not at the event but way long before. If you don't test to restore your backups, you don't have a backup. How does it go unnoticed that S3 backups don't work for so long?
Helpful hint: Have a employee who regularly accidentally deletes folders. I have a couple, it's why I know my backups work. :D
Even better, have a Chaos Monkey do it ;)
Re: GitLab Database Incident – Live Report
#125This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…
Another lesson is this one which I also learned the hard way: don't work long hours or late at night unless it is absolutely necessary. It doesn't sound like it was totally necessary for YP to be pushing this hard, and pushing that hard is what leads to these kinds of errors.
Take your time and work when you're at your best.
Re: GitLab Database Incident – Live Report
#126Seems like very basic mistakes were made, not at the event but way long before. If you don't test to restore your backups, you don't have a backup. How does it go unnoticed that S3 backups don't work for so long?
Yeah, the "You don't have backups unless you can restore them" stikes again. Virtually the only way to lose data is to not have backups. We live in such fancy times that there's no reason to ever lose data that you care about.
Not "can restore them", it's "have restored them".
Best way to ensure that is to have backup restoration be a regularly scheduled event. For most apps I work on, that's either daily or (worst case) weekly, with prod being entirely rebuilt in a lower environment. Works great for creating a test lane too!
Re: GitLab Database Incident – Live Report
#127>1. LVM snapshots are by default only taken once every 24 hours. YP happened to run one manually about 6 hours prior to the outage >2. Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size. >3. SH: It looks like pg_dump may be failing because PostgreSQ…
everything is broken, and we are usually late to notice due to the infrequency of happy-path divergence.
Re: GitLab Database Incident – Live Report
#128Earlier quoted context omitted.
I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.
Nope, that would be me.
All I could think as I read the postmortem was "I hope YP is doing okay". We've all been there, and I hope you're doing okay!
Re: GitLab Database Incident – Live Report
#129 a user for using a repository as some form of CDN, resulting in 47 000 IPs signing in using the same account
47 000 simultaneous users suggests that wouldn't be a small project that did so.
Re: GitLab Database Incident – Live Report
#130I had a great manager a little while back who said they had an expression in Spain:
> "The person who washes the dishes is the one who breaks them."
Not, like, all the time. But sometimes. If you don't have one of these under your belt, you might ask yourself if you're moving too slow.
If that didn't help, he would also point out:
> "This is not a hospital."
Whatever the crisis, and there were some good ones, we weren't going to save anyone's life by running around.
Sure, data loss sucks, but nobody died today because of this.
I really appreciate the raw timeline. I feel your pain. Get some sleep. Tomorrow is a new day.