GitLab Database Incident – Live Report • Hacker News Board

Re: GitLab Database Incident – Live Report

#101

Post by overcast » Wed, Feb 01, 2017, 2:01 AM UTC

Washuu wrote: post #92

ckdarby wrote: post #78

"Our backups to S3 apparently don’t work either: the bucket is empty" 6/6 failed backup procedures. Looks like they are going to be hiring a new sysadmin/devops person...

The best system administrator is the one that has learned from their catastrophic fuck up. To that effect, I still have the same job as I did before I ran "yum update" without knowing it attempts to do in place kernel upgrades. Which resulted in a corrupted RedHat installation on a server we could not turn off.

There is learning from a catastrophic fuck up, and then there is incompetence. Backups is like Day 1, SysAdmin 101. I can't quite grasp how so many different backup systems were left unchecked. Every morning I receive messages saying everything is fine, yet I still go into the backup systems, to make sure they actually did run. In case there was issue with the system alerting me.

Re: GitLab Database Incident – Live Report

#102

Post by grhmc » Wed, Feb 01, 2017, 2:02 AM UTC

sytse wrote: post #42

detaro wrote: post #12

Earlier quoted context omitted.

As much as I appreciate GitLabs extreme openness, that's maybe something that by policy shouldn't be part of published reports. Internal process is one thing, if something goes really bad customers might not be so good at "blameless postmortems" if they have a name to blame.

That is why we went with initials. And I hope customers understand the blame is with all of us, starting with me. Not with the person in the arena. https://twitter.com/sytses/status/826598260831842308

It seems to me that, as a customer, it is blame-shifting away from the company to a particular person. Blameless post-mortems are great, but when speaking to people outside the company I think it is important to own it collectively, "after a second or two we notice we ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com." I believe this isn't your intention, but that is how I interpreted it.

Re: GitLab Database Incident – Live Report

#103

Post by toomuchtodo » Wed, Feb 01, 2017, 2:07 AM UTC

YorickPeterse wrote: post #10

CameronBanga wrote: post #5

Earlier quoted context omitted.

I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.

Nope, that would be me.

Infrastructure ops here. Happens to the best of us. Just have to get back on the horse.

Re: GitLab Database Incident – Live Report

#104

Post by ocdtrekkie » Wed, Feb 01, 2017, 2:07 AM UTC

First of all, I want to say: I think the GitLab people are incredible. They're open and transparent about everything they do, and I think that in a world dominated by GitHub, the fact that GitLab exists is hugely important to decentralization and the continued march of open source. This document continues to demonstrate how the GitLab people are great, transparent people.

That being said, this is why you shouldn't entrust a cloud service to keep your data safe: "So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."

My backups work. I know they work, because I run them and I test them. People entrusting cloud services to have good backups cannot say that.

Re: GitLab Database Incident – Live Report

#105

Post by karmicthreat » Wed, Feb 01, 2017, 2:09 AM UTC

m_fam_wa_k wrote: post #21

eric_h wrote: post #14

Earlier quoted context omitted.

Good lesson on making command prompts on machines always tell you exactly what machine you're working on.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

I've done this exact thing my my servers. Also like GL I prepend PRODUCTION and STAGING on PS1.

I should probably make the PRODUCTION flash just in case.

Re: GitLab Database Incident – Live Report

#106

Post by connorshea » Wed, Feb 01, 2017, 2:10 AM UTC

alyandon wrote: post #90

Thank you for the transparency. This is a good read and I'm going to be sharing it with coworkers tomorrow. :)

Glad to hear others will learn from our mistakes, we certainly are :)

Re: GitLab Database Incident – Live Report

#107

Post by koolba » Wed, Feb 01, 2017, 2:13 AM UTC

mschuster91 wrote: post #54

Walkman wrote: post #25

Seems like very basic mistakes were made, not at the event but way long before. If you don't test to restore your backups, you don't have a backup. How does it go unnoticed that S3 backups don't work for so long?

> How does it go unnoticed that S3 backups don't work for so long? My uneducated guess (this one hit a friend of mine): expired/revoked AWS credentials combined with a backup script that doesn't exit(1) on failure and just writes the exception trace to stderr.

I bet it writes to a log file. It just doesn't alert anyone on failure so the log just grows and grows daily with the same error.

Re: GitLab Database Incident – Live Report

#108

Post by secfirstmd » Wed, Feb 01, 2017, 2:14 AM UTC

nowarninglabel wrote: post #83

gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

Yes, good tip from "Turn the Ship Around" by David Marquet is to use the "I intend to" model. For every action you are going to undertake that is critical, first announce your intentions and give enough time for reactions from others before following through.

"I intend to" is actually a phrased deliberately used a lot during military mission briefings. I always wondered why, I guess it's deliberate.

Re: GitLab Database Incident – Live Report

#109

Post by MattRogish » Wed, Feb 01, 2017, 2:15 AM UTC

gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

We have 2 people on rotation and anything like this you pair with someone else and talk through it. As we like to say "It's only a mistake if we both make it."

Re: GitLab Database Incident – Live Report

#110

Post by [deleted] » Wed, Feb 01, 2017, 2:15 AM UTC

connorshea wrote: post #57

gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

Yup, it's never the fault of a person, always of the system. Once we get this resolved we'll definitely look at ways to prevent anything like it in the future.

[deleted]