Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

581–590 of 621 posts

Re: GitLab Database Incident – Live Report

#581

Post by rhizome »

testUser69 wrote: post #512
YorickPeterse wrote: post #476

Earlier quoted context omitted.

I myself initially added my name to the document in various parts, this was later changed to just initials. I specifically told my colleagues it was OK to keep it in the document. I have no problems taking responsibility for mistakes, and making sure they don't happen ever again.

I don't know why people on hacker news are against transparency. I'm glad you guys are live streaming this, others would feel too inadequate to do so. Being this transparent only makes me want to use (and contribute to) gitlab even more.

I don't know why people on hacker news are against transparency.

It appears to be only a couplefew people on HN who have a real problem with it, so maybe temper your generalizations in a way that they aren't.

Re: GitLab Database Incident – Live Report

#582

Post by zokier »

catwell wrote: post #322
ryandrake wrote: post #185

Earlier quoted context omitted.

If you get the chance to observe pilots operating in the cockpit, I'd recommend it. Every important procedure (even though the pilot has it memorized) is done with a checklist. Important actions are verbally announced and confirmed: "You have the controls" "I have the controls". Much of flight training deals with situational awareness and eliminating distractions in the cockpit. Crew Resource Management[1]. 1: https:…

Great operations teams have incident response procedures based on checklists and clear communication channels like that as well. If this interests you I recommend David Mytton's talk at dotScale 2015: https://www.youtube.com/watch?v=4qGcTOQRvEU

Another talk about using checklists from StrangeLoop 2016:

"Building Secure Cultures" by Leigh Honeywell https://www.youtube.com/watch?v=2BvVZU4IPKc

(checklist part starting from about 16:00)

Re: GitLab Database Incident – Live Report

#583

Post by user15672 »

rhizome wrote: post #580
scaryclam wrote: post #573

Earlier quoted context omitted.

I think perhaps, you want to not do this in the future. Incident reports are about focusing on the "what" and "when" not the "who". This is not about taking responsibility (you don't need to be published on the internet to do that) and you can always have a follow up post after the incident report has been published as a "what I learned during incident X". While it's great you're OK with publishing your name, you've…

I think you're blowing it out of proportion. If you showed it to someone and they told you they assumed it was about fingerpo...Look, it's not that big a deal. They decided to do it, not everything is a blame game.

Woah there! I think you may have misread the parent as it looks like some friendly advice to me (with actual reasons and stuff), rather than the "You shouldn't have done that! You've destroyed your company!!!!" you seem to have read it as.

Heck, they didn't even say to retract anything from the report, just maybe to leave adding names to things until a later date in future incidents.

Re: GitLab Database Incident – Live Report

#584

Post by throwaway91111 »

omouse wrote: post #533
jawilson2 wrote: post #483

Earlier quoted context omitted.

That's awesome, but why publicize it? This isn't an act of contrition for you, no one outside your team really needs to see your dirty laundry, and actually comes off as unprofessional to me. The gitlab team is a team, and you take responsibility as a team. Placing names and initials in the liveblog makes it look like SOMEONE is trying to assign and pass off blame, even if that is not what is happening. Presumably in…

On the contrary, it comes off as very professional. All other companies would hide this, they would show off a very cleaned up post-mortem and say "problem solved" and that's it. Ok so what does that mean, does it mean the process will change for the future or that they just fixed it for today? This is also an awesome advert to see how they work remotely all together and I'm sure they're hiring for DevOps people now…

Naming individuals is not professional. Even allowing it with permission does not set a good standard for operation.

Re: GitLab Database Incident – Live Report

#585

Post by throwaway91111 »

cjbprime wrote: post #496
jawilson2 wrote: post #483

Earlier quoted context omitted.

That's awesome, but why publicize it? This isn't an act of contrition for you, no one outside your team really needs to see your dirty laundry, and actually comes off as unprofessional to me. The gitlab team is a team, and you take responsibility as a team. Placing names and initials in the liveblog makes it look like SOMEONE is trying to assign and pass off blame, even if that is not what is happening. Presumably in…

You're assuming the only reason for wanting to do it would be as contrition, but it sounds like that's not the reason here. Possibly the GitLab team cares about transparency to the extent that they simply prefer to be transparent.

It's not transparent to name people; it's unnecessary and betrays a misunderstanding of how to handle an incident.

Re: GitLab Database Incident – Live Report

#586

Post by js2 »

js2 wrote: post #539

If you're a sys admin long enough, it will eventually happen to you that you'll execute a destructive command on the wrong machine. I'm fortunate that it happened to me very early in my career, and I made two changes in how I work at the suggestion of a wiser SA. 1) Before executing a destructive command, pause. Take your hands off the keyboard and perform a mental check that you're executing the right command on the…

Here's the .bashrc/.bash_login I use:

https://gist.github.com/jaysoffian/8c75e661f7a61b0d094703e26...

Re: GitLab Database Incident – Live Report

#587

Post by GrinningFool »

CrLf wrote: post #415
nanch wrote: post #413

Earlier quoted context omitted.

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…

In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)

Good! I'd like to talk about what the engineer learned from the experience. Certainly if trawling through someone's public repos and records turns up a pattern of repeated mistakes, that should be considered - but the mistakes we all make from time to time are chances to learn.

So what I'd be interested in seeing is if the candidate did learn. The mistake is less important than the candidate demonstrating they moved past it as a stronger developer.

On the flip side - given a choice in situation, I'd prefer not to work for a place that dredges up my old bugs and uses them in isolation as a basis for their decision. That suggests the kind of environment I wouldn't enjoy being in.

Re: GitLab Database Incident – Live Report

#588

Post by rhizome »

user15672 wrote: post #583
rhizome wrote: post #580

Earlier quoted context omitted.

I think you're blowing it out of proportion. If you showed it to someone and they told you they assumed it was about fingerpo...Look, it's not that big a deal. They decided to do it, not everything is a blame game.

Woah there! I think you may have misread the parent as it looks like some friendly advice to me (with actual reasons and stuff), rather than the "You shouldn't have done that! You've destroyed your company!!!!" you seem to have read it as. Heck, they didn't even say to retract anything from the report, just maybe to leave adding names to things until a later date in future incidents.

I'm not sure impugning their professionalism qualifies as friendly advice.

Re: GitLab Database Incident – Live Report

#589

Post by willemmali »

wodny wrote: post #457
kstrauser wrote: post #150

Earlier quoted context omitted.

BTW, find ... -delete avoids any potential shell escaping weirdness and saves you a fork() per file.

This seems to be the best here. As a side note: if someone does something more complicated and uses piping find output to xargs, there are very important arguments to find and xargs to delimit names with binary zero -- -print0 and -0 respectively. Very interesting article: https://www.dwheeler.com/essays/fixing-unix-linux-filenames.... .

I've been writing an `sh`-based tool to check up on my local Git repos, and it uses \0-delimited paths and a lot of `find -print0` + `xargs -0`:

https://gitlab.com/willemmali-sh/chegit/blob/master/chegit#L...

I admit the code can look a little weird, but it was because I had some rather tight contrainst: 1 file, all filenames `\0` separated internally and just POSIX `sh`. I still wanted to reuse code and properly quote variables inside `xargs` invocations (because `sh` does not support `\0`-separated read's), so I ended up having to basically paste function definitions into strings and use some fairly expansive quotation sequences.

Re: GitLab Database Incident – Live Report

#590

Post by tcpipcowboy »

I'm curious to know what strategy has been developed out of this regarding delivery of spam through creation of snippets. In the original GitLab First Incident report it noted "spammers were hammering the database by creating snippets, making it unstable". So many easily accessible platforms are out there that this method of spamming could be used on that it seems like a necessity to evaluate current workflows and identify where checks/balances can be inserted that would prevent this from happening again. Short of removal of snippets, there must be some method of snippet grepping that would put a pause on suspicious snippets, preventing the bulk of submissions along the lines of what GitLab initially received.
Post reply on HN