Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

231–240 of 621 posts

Re: GitLab Database Incident – Live Report

#231

Post by misiti3780 »

ryandrake wrote: post #185
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

If you get the chance to observe pilots operating in the cockpit, I'd recommend it. Every important procedure (even though the pilot has it memorized) is done with a checklist. Important actions are verbally announced and confirmed: "You have the controls" "I have the controls". Much of flight training deals with situational awareness and eliminating distractions in the cockpit. Crew Resource Management[1]. 1: https:…

As documented in this great book:https://www.amazon.com/Checklist-Manifesto-How-Things-Right/...

not just pilots ... doctors, nurses, etc.

Re: GitLab Database Incident – Live Report

#234

Post by Fuzzwah »

mrbill wrote: post #218
imcoconut wrote: post #198

Earlier quoted context omitted.

do you happen to have the link for that video (and related discussion if posted on HN)?

Maybe https://www.youtube.com/watch?v=8W4cfIyNvts ?

I didn't hear any "I agree" in that one.

I found this one which does:

https://www.youtube.com/watch?v=PAaMuTRGP6k&t=1h7m30s

Re: GitLab Database Incident – Live Report

#235

Post by vhost- »

I haven't seen or done anything of this scale before, but I did have a very sobering moment while working on a large online retailers stack as a systems engineer.

We were rolling out a new stack in another data center across the country and before replication went live, I decided to connect and check things out. Our chef work hadn't completed for the database hosts, so I decided to install some OS updates by hand using pssh on all the MySQL hosts and saw a kernel update. So I thought, the DC isn't live yet, no replication is running, I'll just restart these servers. So I did using pssh again and then I caught a glimps at the domain in some output and my face went completely pale. I restarted the production databases... all of them. And they all had 256GB of ecc memory. It takes a very long time for each of those machines to POST.

I contacted the client and said the maintenance page was my fault and was fully expecting to be fired on the spot, but they just grilled me about being careful in the future, and then laughed it off.

I've been the most careful ever since then. It scared me straight. Always make sure you are in the right environment before you do anything that requires a write operation.

Re: GitLab Database Incident – Live Report

#236

Post by Tassels »

sbuttgereit wrote: post #8
CameronBanga wrote: post #2

Not sure if the doc here is refreshing or scary. But Godspeed GitLab team. I've loved the product for about two years now, so curious to see how this plays out.

It's both. I very much appreciate their forthrightness and the way they conduct their company generally. Having said that, I have the code I work on, related content, and a number of clients on the service. [edit for additional point] They need the infrastructure guy they've been looking for sooner than later. I hope there's good progress on that front.

You've got working backups though, right? Or at least a local copy?

Re: GitLab Database Incident – Live Report

#237

Post by throwaway3347 »

hueving wrote: post #230
throwaway3347 wrote: post #209

Earlier quoted context omitted.

Like you, I would like to add my 2 cent, which I hope will be taken positively, as I would like to see them provide healthy competition for GitHub for years to come. Since GitLab is so transparent about everything, from their marketing/sales/feature proposals/technical issues/etc., they make it glaringly obvious, from time to time, that they lack very fundamental core skills, to do things right/well. In my opinion, t…

Why would they try to recruit from Microsoft? Most of the software engineers at Microsoft are not focused on developing scalable web services architectures. And the ones that do have built up all of their expertise with Microsoft technologies (.net running on Windows server talking to mssql). >Microsoft and others that have top talent/domain expertise, will have the advantage. Again, Microsoft isn't even in this same…

> Why would they try to recruit from Microsoft?

I'm not sure if you read my post correctly, but I never mentioned poaching from Microsoft. I said compete for programmers that would choose to work for Microsoft. I'm also not sure if you understand what Microsoft does. It's a very diverse company with R&D spending that rivals some small nations.

> Microsoft isn't even in this same field (git hosting

I guess you haven't heard of https://www.visualstudio.com/team-services/ and their on premise TFS solution that supports Git.

Microsoft understands Enterprise and it's quite obvious they want to be a major provider for Git hosting. It will be foolish to believe Microsoft is not focused on owning the Git mindshare in Enterprise.

> Are you an employee there or something

No. Just somebody that understands this problem space.

Re: GitLab Database Incident – Live Report

#239

Post by wereHamster »

connorshea wrote: post #57
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

Yup, it's never the fault of a person, always of the system. Once we get this resolved we'll definitely look at ways to prevent anything like it in the future.

At some point you have to blame the person. If the person did it wilfully, deliberately, ignoring the checks and safeguards. Or because there's already so much system that having any more of it would place severe restrictions on everyday tasks.

I'm not saying that's the case here, because it does seem that GitLab has systemic deficiencies. But "never" and "always" are such strong statements.

Prosecutor: Mr. Accused, here is evidence that you murdered that other person. Accused: It's not my fault, but the systems'. Judge: Oh, ok. You are a free man.

Re: GitLab Database Incident – Live Report

#240

Post by jjirsa »

hobarrera wrote: post #134
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.

And then your disk fills because your new hire doesn't know about trash and you have an outage and remove it
Post reply on HN