Postmortem of database outage of January 31 • Hacker News Board

Re: Postmortem of database outage of January 31

#71

Post by ronack » Sat, Feb 11, 2017, 2:16 AM UTC

MichaelGG wrote: post #22

bitshepherd wrote: post #17

Earlier quoted context omitted.

A great number of issues can be attributed to the selection of Azure as the platform of choice. That said, a little bird told me that the decision was largely a cost factor. "You get what you pay for" never rang more true.

Is Azure cutting deals beyond the usual free 60k over a year or two or whatever it is for cool startups? Azure seems significantly more expensive in general, problems and slowness aside.

Azure offers $500k for YC startups and $360k for "lesser" accelerators. It's by far the most generous offer out there.

Re: Postmortem of database outage of January 31

#72

Post by carterehsmith » Sat, Feb 11, 2017, 2:18 AM UTC

tschellenbach wrote: post #45

Shouldn't the conclusion of this post mortem be a move to a managed database service like RDS? The database doesn't sound huge, RDS is affordable enough, sounds to me that you spend less money and have better uptime and sleep by moving away from this in-house solution.

According to this article:

> https://www.theregister.co.uk/2016/11/14/gitlab_to_dump_clou...

...they were moving away from the cloud, to their own servers.

Re: Postmortem of database outage of January 31

#73

Post by ronack » Sat, Feb 11, 2017, 2:18 AM UTC

bitshepherd wrote: post #17

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

A great number of issues can be attributed to the selection of Azure as the platform of choice. That said, a little bird told me that the decision was largely a cost factor. "You get what you pay for" never rang more true.

Yes, I recall seeing a ticket that referenced Gitlab using Azure because it was heavily subsidized. My company uses Azure for much the same reason, and my experience has been largely positive.

Re: Postmortem of database outage of January 31

#74

Post by sytse » Sat, Feb 11, 2017, 2:21 AM UTC

carterehsmith wrote: post #72

tschellenbach wrote: post #45

Shouldn't the conclusion of this post mortem be a move to a managed database service like RDS? The database doesn't sound huge, RDS is affordable enough, sounds to me that you spend less money and have better uptime and sleep by moving away from this in-house solution.

According to this article: > https://www.theregister.co.uk/2016/11/14/gitlab_to_dump_clou... ...they were moving away from the cloud, to their own servers.

We changed our minds. We're working on a post in https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/... but we wanted to post this first.

Re: Postmortem of database outage of January 31

#75

Post by bdcravens » Sat, Feb 11, 2017, 2:22 AM UTC

imperialdrive wrote: post #57

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

I'm typing off the top of my head, but didn't they have like 400GB of database? That would probably take 27 hours to get fully available via S3 at 32,000/kbps which is about what s3 will provide for first time hits in my experience.

RDS can restore a point-in-time snapshot in a couple of hours on databases over 1TB (speaking from experience)

Re: Postmortem of database outage of January 31

#76

Post by carterehsmith » Sat, Feb 11, 2017, 2:35 AM UTC

sytse wrote: post #74

carterehsmith wrote: post #72

Earlier quoted context omitted.

According to this article: > https://www.theregister.co.uk/2016/11/14/gitlab_to_dump_clou... ...they were moving away from the cloud, to their own servers.

We changed our minds. We're working on a post in https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/... but we wanted to post this first.

It would be great for all of us if you provided your reasoning: to cloud, or not to cloud :) Thanks

Re: Postmortem of database outage of January 31

#77

Post by renaudg » Sat, Feb 11, 2017, 2:36 AM UTC

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

This is usually true, except when it's not :

I have personally experienced a near-catastrophic situation 3 years ago, where 13 out of 15 days' worth of nightly RDS MySQL snapshots were corrupt and would not restore properly.

The root cause was a silent EBS data corruption bug (RDS is EBS-based), that Amazon support eventually admitted to us had slipped through and affected a "small" number of customers. Unlucky us.

We were given exceptional support including rare access to AWS engineers working on the issue, but at the end of the day, there was no other solution than to attempt restoring each nightly snapshot one after the other, until we hopefully found one that was free of table corruption. The lack of flexibility to do any "creative" problem-solving operations within RDS certainly bogged us down.

With a multi-hundred gigabyte database, the process was nerve-wracking as each restore attempt took hours to perform, and each failure meant saying goodbye to another day's worth of user data, with the looming armageddon scenario that eventually we would reach the end of our snapshots without having found a good one.

Finally, after a couple of days of complete downtime, the second to last snapshot worked (IIRC) and we went back online with almost two weeks of data loss, on a mostly user-generated content site.

We got a shitload of AWS credits for our trouble, but the company obviously went through a very near-death experience, and to this day I still don't 100% trust cloud backups unless we also have a local copy created regularly.

Re: Postmortem of database outage of January 31

#78

Post by gumby » Sat, Feb 11, 2017, 2:37 AM UTC

NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

> Mark my words, the board members from the VC firms will be removed by the VC partners

I have never, ever, seen this. Every firm has its own internal politics [1] but rarely if ever would they do this. They are more likely to just ignore it.

There is a belief that it takes a decade to tell if a VC is any good or not, and that includes "learning experiences" (all on the LP's dime of course).

> Then VC firms will put an experienced CEO and CTO in place

Now this I have seen. It even works sometimes (e.g. Eric Schmidt/Google).

[1] I have had a firm invest in which all decisions were made by a single partner. I also had a firm invest (a sizable sum!) in which other senior partners never met me and only learned what my company even did when I gave a presentation at one of their LP meetings. Also some very large funds allow senior partners to make small seed investments ("science projects") without formal approval from the partnership.

Re: Postmortem of database outage of January 31

#79

Post by koolba » Sat, Feb 11, 2017, 2:42 AM UTC

renaudg wrote: post #77

illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

This is usually true, except when it's not : I have personally experienced a near-catastrophic situation 3 years ago, where 13 out of 15 days' worth of nightly RDS MySQL snapshots were corrupt and would not restore properly. The root cause was a silent EBS data corruption bug (RDS is EBS-based), that Amazon support eventually admitted to us had slipped through and affected a "small" number of customers. Unlucky us. W…

> We got a shitload of AWS credits for our trouble, but the company obviously went through a very near-death experience, and to this day I don't 100% trust cloud backups unless we also have a local copy created regularly.

Cloud backups, and more generally all backups should be treated like nuclear profliferation treaties: Trust, but verify!

If your periodically restore your backups you'll catch this kind of crap when it's not an issue, rather than when shit had already hit the fan.

Re: Postmortem of database outage of January 31

#80

Post by voidlogic » Sat, Feb 11, 2017, 2:45 AM UTC

>When we went to look for the pg_dump backups we found out they were not there. The S3 bucket was empty, and there was no recent backup to be found anywhere. Upon closer inspection we found out that the backup procedure was using pg_dump 9.2, while our database is running PostgreSQL 9.6 (for Postgres, 9.x releases are considered major). A difference in major versions results in pg_dump producing an error, terminating the backup procedure.

Yikes. One common practice that would have avoided this is by using the just taken backup to populate stage. If the restore fails pages go out. If integration tests that run after a successful restore/populate fail- pages go out.

Live and learn I guess.