Postmortem of database outage of January 31 • Hacker News Board

Re: Postmortem of database outage of January 31

#81

Post by voidlogic » Sat, Feb 11, 2017, 2:47 AM UTC

koolba wrote: post #79

renaudg wrote: post #77

Earlier quoted context omitted.

This is usually true, except when it's not : I have personally experienced a near-catastrophic situation 3 years ago, where 13 out of 15 days' worth of nightly RDS MySQL snapshots were corrupt and would not restore properly. The root cause was a silent EBS data corruption bug (RDS is EBS-based), that Amazon support eventually admitted to us had slipped through and affected a "small" number of customers. Unlucky us. W…

> We got a shitload of AWS credits for our trouble, but the company obviously went through a very near-death experience, and to this day I don't 100% trust cloud backups unless we also have a local copy created regularly. Cloud backups, and more generally all backups should be treated like nuclear profliferation treaties: Trust, but verify! If your periodically restore your backups you'll catch this kind of crap when…

I agree, the backup that isn't used to populate a stage/qa instance right after being taken is untrushworthy.

Re: Postmortem of database outage of January 31

#82

Post by chrismorgan » Sat, Feb 11, 2017, 2:50 AM UTC

meowface wrote: post #10

>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from readin…

I messed up an XP computer at home with `cd D:\backups\something; del /s * ` many years ago; `cd` without the /D flag doesn’t change the drive, so although D:\backups\something was the working directory on the D: drive, the working directory was still C:\WINDOWS\system32, and cmd.exe was running as administrator.

Fortunately disks were slower back then, so it hadn’t deleted too many files when I interrupted it, and the computer was able to be recovered without too much inconvenience.

Re: Postmortem of database outage of January 31

#83

Post by koolba » Sat, Feb 11, 2017, 2:54 AM UTC

voidlogic wrote: post #81

koolba wrote: post #79

Earlier quoted context omitted.

> We got a shitload of AWS credits for our trouble, but the company obviously went through a very near-death experience, and to this day I don't 100% trust cloud backups unless we also have a local copy created regularly. Cloud backups, and more generally all backups should be treated like nuclear profliferation treaties: Trust, but verify! If your periodically restore your backups you'll catch this kind of crap when…

I agree, the backup that isn't used to populate a stage/qa instance right after being taken is untrushworthy.

Even before that there are steps you can take. For example if you take a Postgres backup with pg_dump, you can run pg_restore on it to verify it.

If a database isn't specified, pg_restore will output the SQL commands to restore the database and the exit code will be zero (success) if it makes it through the entire backup. That lets you know that the original dump succeeded and there was no disk error for whatever was written. Save the file to something like S3 as well as the sha256 of it. If the hash matches after you retrieve you can be pretty damn sure that it's a valid backup!

Otherwise you get the blind scripts like GitLab had where pg_dump fails. No exit code checking. No verification. No beuno!

Re: Postmortem of database outage of January 31

#84

Post by [deleted] » Sat, Feb 11, 2017, 2:55 AM UTC

NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

[deleted]

Re: Postmortem of database outage of January 31

#85

Post by elliotec » Sat, Feb 11, 2017, 2:59 AM UTC

NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

They also openly pay well below market rate, so they should expect to get what they pay for.

Re: Postmortem of database outage of January 31

#86

Post by ebiester » Sat, Feb 11, 2017, 3:02 AM UTC

NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

I am a senior engineer who has seen shit go down. I am quite literally the graybeard.

I am not a GitLab customer, I am not a startup junkie, and I'm usually considered one of the more conservative (in action, not politics) engineers in my peer group in technology adoption.

The cloud is just someone else's computer.

However, I've also seen graybeards who should have known better fuck something up. I've seen a team of smart people who in a moment of crisis made the wrong decision. I am currently in an organization that is full of careful people and have still seen data loss.

I went trawling through LinkedIn for GitLab employees, and they certainly have their fair share of senior engineers. If you want to fault them for being a remote company, that's fine, but is it that different than a fortune 500 company that has developers in the Bay Area, Austin, India, China, Budapest, and remote workers in other locations?

Or is a company only legitimate if it's in an open space in the Valley?

Re: Postmortem of database outage of January 31

#87

Post by AlisdairO » Sat, Feb 11, 2017, 3:21 AM UTC

yeukhon wrote: post #66

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

I will add my inputs on RDS. I gave this comment on the GitLab incident thread. I actually managed to delete an RDS cloudformation stack by accident. The night before I pushed an update to Cloudformation to convert storage class to provisioned IOPS. Next morning I woke up really early, drove my girlfriend to work. While waiting in the car I wanted to check the status of the update so I went on AWS mobile app to check…

Not sure how it works on cloudformation, but in the console and API you have to explicitly skip the final snapshot.

Re: Postmortem of database outage of January 31

#88

Post by p4lindromica » Sat, Feb 11, 2017, 3:29 AM UTC

NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

Exactly. The technical problems are not the root cause. How did the existing processes (or lack thereof) fail? How did the organization fail?

Re: Postmortem of database outage of January 31

#89

Post by gech » Sat, Feb 11, 2017, 3:29 AM UTC

NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

Didn't Samsung have some phones catch on fire? Didn't Delta go down recently?

Re: Postmortem of database outage of January 31

#90

Post by bogomipz » Sat, Feb 11, 2017, 3:43 AM UTC

elliotec wrote: post #85

NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

They also openly pay well below market rate, so they should expect to get what they pay for.

Can you elaborate on that?

For instance I just looked at their jobs listings and they advertise a range for annual compensation, the high end of that range looks about right for the locations I checked SF, and London in the UK for PE positions.

https://about.gitlab.com/jobs/production-engineer/