Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Postmortem of database outage of January 31

about.gitlab.com

Post reply on HN
Print view View original

111–120 of 269 posts

Re: Postmortem of database outage of January 31

#111

Post by pzh »

pfarnsworth wrote: post #102
NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

At a startup I worked at, the VP of Engineering was brilliant. He was probably the smartest person I've ever worked for, and the most hard working. He was online almost all hours of the day, working. He also insisted on a 9-to-5 schedule for all the engineers, because he believed that killing your engineers with work was not a scalable way to build a team. He was great. But the first month I was there, I kept pressin…

Maybe in his own cocky way, his disaster recovery plan was to be smart enough to never have a disaster. The problem with this approach is that it only works for a small company, and many people figure out the hard way that smarts don't scale.

Re: Postmortem of database outage of January 31

#112

Post by developer2 »

NPegasus wrote: post #63

> Root Cause Analysis > [...] > [List of technical problems] No, the root cause is you have no senior engineers who have been through this before. A collection of distributed remote employees, none of whom has enough experience to know any of the list of "Basic Knowledge Needed to Run a Website at Scale" that you list as the root causes. $30 million in funding and still running the company like a hobby project among…

>> the root cause is you have no senior engineers who have been through this before

They openly publish their database hostnames in this postmortem (db1.cluster.gitlab.com and db2.cluster.gitlab.com). These actually have public DNS that resolves. The last straw: port 22 on each server is running an open sshd server (the fact that password auth is disabled is of little consolation).

A production database server should NEVER HAVE a public IP address to start with. This is simply unacceptable and proves they don't have a single person qualified to handle infrastructure. Their only concern is that their developers can ssh into every production server without having to deal with vpns or firewalls.

Huge red flag that your data cannot be trusted.

Re: Postmortem of database outage of January 31

#113

Post by sytse »

grzm wrote: post #110
pzh wrote: post #108

Earlier quoted context omitted.

I know somebody who knows the guy who deleted the data that caused the outage, and in his words (the guy who knows the GitLab employee), the GitLab engineer is one of the smartest people he's ever known--in fact, he's actually brilliant. So you can rest assured that the data loss wasn't caused by a an inexperienced kid.

You're aware he's in this thread, right? And the hearsay matches up with his comments, from what I've read.

Yes, he's one of our smartest people.

Re: Postmortem of database outage of January 31

#114

Post by chuckdries »

cjbprime wrote: post #51
rectang wrote: post #20

Earlier quoted context omitted.

For me, it was when I meant to execute this... rm -rf ~/foo ... but executed this instead: rm -rf ~ /foo

My worst data loss was: $ tar cvfz mbox outbox mbox.tar.gz The argument order is backwards -- the output file is supposed to be first, then the input files. On my system, this overwrote my full mailbox with a gzipped copy of my outbox, and a complaint that the mbox.tar.gz input file didn't exist. That's right, the worst data loss happened while I was trying to take a backup. :(

What were you trying to do? What's the outbox argument for?

Re: Postmortem of database outage of January 31

#115

Post by [deleted] »

cjbprime wrote: post #51
rectang wrote: post #20

Earlier quoted context omitted.

For me, it was when I meant to execute this... rm -rf ~/foo ... but executed this instead: rm -rf ~ /foo

My worst data loss was: $ tar cvfz mbox outbox mbox.tar.gz The argument order is backwards -- the output file is supposed to be first, then the input files. On my system, this overwrote my full mailbox with a gzipped copy of my outbox, and a complaint that the mbox.tar.gz input file didn't exist. That's right, the worst data loss happened while I was trying to take a backup. :(

[deleted]

Re: Postmortem of database outage of January 31

#116

Post by lojack »

bogomipz wrote: post #90
elliotec wrote: post #85

Earlier quoted context omitted.

They also openly pay well below market rate, so they should expect to get what they pay for.

Can you elaborate on that? For instance I just looked at their jobs listings and they advertise a range for annual compensation, the high end of that range looks about right for the locations I checked SF, and London in the UK for PE positions. https://about.gitlab.com/jobs/production-engineer/

Maxed out everything in my city (Cleveland) before I was able to reach a salary I'd consider, and I'm hardly a Senior engineer.

I wouldn't call their salary range a spit in the face, but its probably around $20k below market rates where I live. Benefits are competitive in my opinion.

Re: Postmortem of database outage of January 31

#117

Post by CaptSpify »

yarper wrote: post #31
jdavis703 wrote: post #19

Earlier quoted context omitted.

Do you expect management to be staring over your shoulder every time you do some kind of `rm` on a production server? With great power comes great responsibility.

People get tired, sick, frustrated, panic - part of being a responsible engineer is accepting you're as fallible as the next person and building in protection against your own errors. However, if "the engineer" that caused this happens to read this, the above is not a sign that you should quit the profession and become a hermit. A chain of events caused this, you just happened to be the one without a chair to sit in…

One of the coolest things I've read about is how airlines do root cause analysis. If you get to a point where a human can mess up the situation like this, it's considered a systemic issue. Mobile now, but can try to find it later

EDIT: https://dvikan.no/ntnu-studentserver/reports/A%20Human%20Err...

> That is, much like falling dominoes, Bird and others (Adams, 1976; Weaver, 1971) have described the cascading nature of human error beginning Human Error Perspective 39 with the failure of management to control losses (not necessarily of the monetary sort) within the organization.

Re: Postmortem of database outage of January 31

#118

Post by lojack »

p4lindromica wrote: post #98
bogomipz wrote: post #90

Earlier quoted context omitted.

Can you elaborate on that? For instance I just looked at their jobs listings and they advertise a range for annual compensation, the high end of that range looks about right for the locations I checked SF, and London in the UK for PE positions. https://about.gitlab.com/jobs/production-engineer/

Compensation is more than just salary

Read through their benefits, they are competitive, but nowhere near enough to make up for the difference in salary.

Re: Postmortem of database outage of January 31

#119

Post by ploxiln »

nodesocket wrote: post #95

My main question is still: >> Why did replication stop? - A spike in database load caused the database replication process to stop. This was due to the primary removing WAL segments before the secondary could replicate them. Is this a bug/defect in PostgreSQL then? Incorrect PostgreSQL configuration? Insufficient hardware? What was the root cause of Postgres primary removing the WAL segments?

There's always some limit. At $PREVIOUS_JOB I think it was at least 48 hours, probably over 72 hours, of replication log retention (usually measured in GiB though). So it's surprising that in GitLab's case it must have been less than 6 hours (IIRC from the original google doc the slave had more than 4 hours replication lag due to load, initially ...)

Re: Postmortem of database outage of January 31

#120

Post by bogomipz »

elliotec wrote: post #100
bogomipz wrote: post #90

Earlier quoted context omitted.

Can you elaborate on that? For instance I just looked at their jobs listings and they advertise a range for annual compensation, the high end of that range looks about right for the locations I checked SF, and London in the UK for PE positions. https://about.gitlab.com/jobs/production-engineer/

Using that same job calculator that you used (for the developer position) near the bottom of this page: https://about.gitlab.com/jobs/developer/ , the rates are way way off from my area. They are out of their minds if they think the top rate for a Senior engineer in Salt Lake City with above average experience is $78k. Most other areas also seem pretty low from what I've looked into but I suppose a few areas could be…

>It's a bit strange that they cut or raise your pay by that calculator when you move cities."

That is odd. Usually if you make more based on your previous location a company won't actually claw anything back you just will likely not be getting any raises.

Post reply on HN