Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

Post reply on HN
Print view View original

121–130 of 269 posts

Re: Postmortem of database outage of January 31

#121

Post by angry_octet »

snackai wrote: post #61
nilved wrote: post #24

Earlier quoted context omitted.

RDS also comes with its own set of tradeoffs. There is no free lunch, and the cloud is just another word for someone else's server. There are reasons Gitlab opposes that.

You are completely right. There are reasons to oppose the cloud, but maybe they should focus on improving their systems before moving out of the cloud. At this point in time it is clear that GitLab lacks the talent to run everything themselves. I mean 5 backups worthless or lost? You can't let interns write your backups system. After all backup is a large portion of their product.

The worst part of the whole episode, even worse than 'deleted the active database by accident', was '(backups) were no one's responsibility'. This is not an oversight by an individual engineer, but an aspect of the management and company culture. It shows they lack processes derived from requirements. Lots on introspection required from gitlab at this point.

Re: Postmortem of database outage of January 31

#122

Post by _Codemonkeyism »

RhodesianHunter wrote: post #99
memracom wrote: post #93

Earlier quoted context omitted.

RDS PostgreSQL is like the Hotel California, you can check in any time you want, but you can never leave. Maybe it is OK as a simple data store for a single app, but not for a real database. I gained a lot of my knowledge of PostgreSQL internals by helping my company get off of RDS and onto a dedicated EC2 instance solution. RDS imposes too many limitations. Also, your snapshot backup solution is trivial to implement…

"Maybe it is OK as a simple data store for a single app, but not for a real database." Currently working at company number 2 with large (many terabytes) databases on RDS and can safely say this is horse shit. The amount of time and energy it allows our engineers to spend on our actual products instead of database management is worth all of the extra cost and lock in and then some. Edit: I just realized that you were…

I'm interested in

a. How many hours would you guess you are saving a month?

b. What takes by a factor of 10 less time on RDS than doing it by hand? What task sees the largest time saved?

Because I always wonder when reading this, what am I missing? What haven't we done? Were we lucky? We were running MySQL and Postgres for multi hundred million EUR companies with millions of users and we did not spend a lot of effort into managing them.

Re: Postmortem of database outage of January 31

#123

Post by [deleted] »

grzm wrote: post #50
cjbprime wrote: post #48

Earlier quoted context omitted.

That said, it did take AWS a disturbingly long time to add support for Postgres read replicas.

Why disturbingly long? As opposed to frustratingly long? Or something else? What does the time it took Amazon to add such support imply to you?

[deleted]

Re: Postmortem of database outage of January 31

#124

Post by lojack »

meowface wrote: post #10

>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from readin…

I'm no sysadmin, and I know mistakes are inevitable and all... but I find this kind of mistake is unlikely to come from me. I feel as though a lot of developers are too nonchalant about production boxes. I think one or two close calls where I nearly did this exact thing served as a good wakeup call for me.

Steps I personally take to avoid this:

- Avoid prod boxes like the plague

- Set up a prompt (globally) to make it extremely obvious that you're in production. Something like a red background and black text saying "PRODUCTION"

- When changing data in production (DB's, config, etc) write a script (or just commands to copy and paste) and have that peer reviewed. If anything doesn't go to plan, treat it as a red flag. This serves a dual purpose of having a quick record of your actions without hunting through logs.

- Never ever leave open sessions

- Avoid prod boxes. This is important enough for me to say twice. Most of the time it can be avoided, especially if you use configuration management tools and write tools to perform common operations.

Now, lets just cross my fingers I don't jinx myself :-)

Re: Postmortem of database outage of January 31

#125

Post by manojlds »

angry_octet wrote: post #121
snackai wrote: post #61

Earlier quoted context omitted.

You are completely right. There are reasons to oppose the cloud, but maybe they should focus on improving their systems before moving out of the cloud. At this point in time it is clear that GitLab lacks the talent to run everything themselves. I mean 5 backups worthless or lost? You can't let interns write your backups system. After all backup is a large portion of their product.

The worst part of the whole episode, even worse than 'deleted the active database by accident', was '(backups) were no one's responsibility'. This is not an oversight by an individual engineer, but an aspect of the management and company culture. It shows they lack processes derived from requirements. Lots on introspection required from gitlab at this point.

That's why I was critical of the other article on HN recently that evangelized their remote work.

Re: Postmortem of database outage of January 31

#126

Post by mikiem »

renaudg wrote: post #77
illumin8 wrote: post #13

I have to say - if they were using a managed relational database service, like Amazon's RDS Postgres, this likely would have never happened. RDS fully automates nightly database snapshots, and ships archive logs to S3 every 5 minutes, which gives you the ability to restore your database to any point in time within the last 35 days, down to the second. Also, RDS gives you a synchronously replicated standby database, a…

This is usually true, except when it's not : I have personally experienced a near-catastrophic situation 3 years ago, where 13 out of 15 days' worth of nightly RDS MySQL snapshots were corrupt and would not restore properly. The root cause was a silent EBS data corruption bug (RDS is EBS-based), that Amazon support eventually admitted to us had slipped through and affected a "small" number of customers. Unlucky us. W…

Snapshots are not backups, although many people use them as backups and believe they are good backups. Snapshots are snapshots. Only backups are backups.

Re: Postmortem of database outage of January 31

#127

Post by sytse »

carterehsmith wrote: post #76
sytse wrote: post #74

Earlier quoted context omitted.

We changed our minds. We're working on a post in https://gitlab.com/gitlab-com/www-gitlab-com/merge_requests/... but we wanted to post this first.

It would be great for all of us if you provided your reasoning: to cloud, or not to cloud :) Thanks

The thing I linked contains a draft blog post with our reasons. But feel free to ask questions after reading that.

Re: Postmortem of database outage of January 31

#128

Post by angry_octet »

pfarnsworth wrote: post #106
yarper wrote: post #14

It's amazing how quickly it descends into "one of the engineers" did x or y. Who was steering this ship exactly? It's really simple to point the finger and try to find a single cause of failure - but it's a fools errand - comparable to finding the single source behind a great success.

90% of all outages are caused by human error. That's why change management solutions were so big 10 years ago, trying to get rid of the human element of changes in an enterprise.

They way to deal with that is not to crush the people, but to analyse the ways human decision making fails and to provide assistance. More often than not the failures occur because the people had not been trained correctly, which is a management problem not a PEBKAC one (how many junior systems people have been trained in how to recover from cascading disk/network/database failures?). ITIL is just a tool for ensuring failures which no one understands because understanding has been devalued.

Re: Postmortem of database outage of January 31

#129

Post by debaserab2 »

mikiem wrote: post #126
renaudg wrote: post #77

Earlier quoted context omitted.

This is usually true, except when it's not : I have personally experienced a near-catastrophic situation 3 years ago, where 13 out of 15 days' worth of nightly RDS MySQL snapshots were corrupt and would not restore properly. The root cause was a silent EBS data corruption bug (RDS is EBS-based), that Amazon support eventually admitted to us had slipped through and affected a "small" number of customers. Unlucky us. W…

Snapshots are not backups, although many people use them as backups and believe they are good backups. Snapshots are snapshots. Only backups are backups.

What is the difference exactly?

Re: Postmortem of database outage of January 31

#130

Post by [deleted] »

lojack wrote: post #124
meowface wrote: post #10

>Trying to restore the replication process, an engineer proceeds to wipe the PostgreSQL database directory, errantly thinking they were doing so on the secondary. Unfortunately this process was executed on the primary instead. The engineer terminated the process a second or two after noticing their mistake, but at this point around 300 GB of data had already been removed. I could feel the sweat drops just from readin…

I'm no sysadmin, and I know mistakes are inevitable and all... but I find this kind of mistake is unlikely to come from me. I feel as though a lot of developers are too nonchalant about production boxes. I think one or two close calls where I nearly did this exact thing served as a good wakeup call for me. Steps I personally take to avoid this: - Avoid prod boxes like the plague - Set up a prompt (globally) to make i…

[deleted]
Post reply on HN