GitLab Database Incident – Live Report • Hacker News Board

Re: GitLab Database Incident – Live Report

#151

Post by omgdresser » Wed, Feb 01, 2017, 3:08 AM UTC

jpgvm wrote: post #144

overcast wrote: post #74

Earlier quoted context omitted.

Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…

5 interviews isn't -that- insane, it's on the high end though. But I do know for sure they have turned down extremely technically gifted people for "non-technical" reasons, the kind of people that would ensure this sort of disaster wouldn't happen. So maybe the process really isn't that great.

I'm biased because I went through the process and was rejected, but I agree. There was more focus on asinine 'behaviorial' questions than the real reason you're talking to someone: are you (relatively) sane, and more importantly are you capable, can you do the job? Everything else is secondary, in my opinion.

Re: GitLab Database Incident – Live Report

#152

Post by stochasticegg » Wed, Feb 01, 2017, 3:08 AM UTC

leesalminen wrote: post #17

This is the stuff my nightmares consist of after 900 consecutive days of being on call (and counting).

I've been on call for nearly 12 years now. My SO and I feel your pain.

Re: GitLab Database Incident – Live Report

#153

Post by DanielDent » Wed, Feb 01, 2017, 3:09 AM UTC

I'm a huge Gitlab fan. But I long ago lost faith in their ability to run a production service at scale.

Nothing important of mine is allowed to live exclusively on Gitlab.com.

It seems like they are just growing too fast for their level of investment in their production environment.

One of the only reasons I was comfortable using Gitlab.com in the first place was because I knew I could migrate off it without too much disruption if I needed to (yay open source!). Which I ended up forced to do on short notice when their CI system became unusable for people who use their own runners (overloaded system + an architecture which uses a database as a queue. ouch.).

Which put an end to what seemed like constant performance issues. It was overdue, and made me sleep well about things like backups :).

A while back one of their database clusters went into split brain mode, which I could tell as an outsider pretty quickly... but for those on the inside, it took them a while before they figured it out. My tweet on the subject ended up helping document when the problem had started.

If they are going to continue offering Gitlab.com I think they need to seriously invest in their talent. Even with highly skilled folks doing things efficiently, at some point you just need more people to keep up with all the things that need to be done. I know it's a hard skillset to recruit for - us devopish types are both quite costly and quite rare - but I think operating the service as they do today seriously tarnishes the Gitlab brand.

I don't like writing things like this because I know it can be hard to hear/demoralizing. But it's genuine feedback that, taken in the kind spirit is intended, will hopefully be helpful to the Gitlab team.

Re: GitLab Database Incident – Live Report

#154

Post by nummy » Wed, Feb 01, 2017, 3:11 AM UTC

6 hour rewind is really not that bad, even if it was 24 hours, that's still really good for the scope.

Re: GitLab Database Incident – Live Report

#155

Post by overcast » Wed, Feb 01, 2017, 3:13 AM UTC

omgdresser wrote: post #147

overcast wrote: post #74

Earlier quoted context omitted.

Apparently the following insane interviewing process, wasn't enough to find someone competent enough to cover the basics. https://about.gitlab.com/jobs/production-engineer/ ------------------- Applicants for this position can expect the hiring process to follow the order below. Please keep in mind that applicants can be declined from the position at any stage of the process. To learn more about someone who may be con…

I went through all of the insane interviews except the CEO interview for the production engineer position and was rejected. I brought quite a bit of high-level operational experience from a few different places, including some pretty high-traffic video sites, HPC and a lucrative e-commerce site. The reason for my rejection? I had an ongoing side project and development/consulting gig which had been paying the bills f…

That's just crazy. Thanks for the insight!

Re: GitLab Database Incident – Live Report

#156

Post by [deleted] » Wed, Feb 01, 2017, 3:13 AM UTC

chrissnell wrote: post #143

MattRogish wrote: post #109

Earlier quoted context omitted.

We have 2 people on rotation and anything like this you pair with someone else and talk through it. As we like to say "It's only a mistake if we both make it."

Absolutely. Same goes for any ops response. You need two people: one to triage the issue and another one to communicate with external stakeholders and to help the one doing the triage. The military does a very similar thing. An Army company commander usually has a RTO (radiotelephone operator) to handle taking on the radio. This frees the commander to make real-time decisions and response quickly to the situation on…

[deleted]

Re: GitLab Database Incident – Live Report

#157

Post by Raticide » Wed, Feb 01, 2017, 3:15 AM UTC

YorickPeterse wrote: post #10

CameronBanga wrote: post #5

Earlier quoted context omitted.

I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.

Nope, that would be me.

It happens to everyone eventually, so don't feel too bad. But at least now you'll have a much more robust backup process.

Re: GitLab Database Incident – Live Report

#158

Post by drewmate » Wed, Feb 01, 2017, 3:15 AM UTC

totally wrote: post #130

I had a great manager a little while back who said they had an expression in Spain: > "The person who washes the dishes is the one who breaks them." Not, like, all the time. But sometimes. If you don't have one of these under your belt, you might ask yourself if you're moving too slow. If that didn't help, he would also point out: > "This is not a hospital." Whatever the crisis, and there were some good ones, we were…

> Get some sleep.

Definitely get sleep, but it would be nice if the site were back online before that. I actually just created a new GitLab account and project a couple days ago for a project I needed to work on with a collaborator tonight. This is not a good first impression.

Re: GitLab Database Incident – Live Report

#159

Post by spydum » Wed, Feb 01, 2017, 3:17 AM UTC

hobarrera wrote: post #146

karmicthreat wrote: post #105

Earlier quoted context omitted.

I've done this exact thing my my servers. Also like GL I prepend PRODUCTION and STAGING on PS1. I should probably make the PRODUCTION flash just in case.

We need a tag for our PS1.

Good news! Enable ANSI in your terminal and: ESC[5m

http://wiki.bash-hackers.org/scripting/terminalcodes

Re: GitLab Database Incident – Live Report

#160

Post by CaliforniaKarl » Wed, Feb 01, 2017, 3:24 AM UTC

First off, my heartfelt commiserations for the GitLab team here. My suggestion: Start watching an hour-long video; the rsync will finish right when it gets to the good part!

I wonder if a future project might be to have the DB-stored stuff use Git as a replication back-end. Like, for example, having each issue be a directory, and individual comments be JSON files. It would never (normally) be the data store "of record" (the DB would), but maybe that would work as a backup?