Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

441–450 of 621 posts

Re: GitLab Database Incident – Live Report

#441

Post by discordianfish »

gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

Checklists are great if you have them and often you should create one even during an outage but other times you waste time. It's really hard to balance fixing things quickly and not breaking more.

There isn't a silver bullet anyway, it's layer on layer of operational best practices what makes you resilient against such issues.

Re: GitLab Database Incident – Live Report

#442

Post by pmlnr »

arc_of_descent wrote: post #400

Don't prepare a backup plan. Prepare a recovery plan.

I agree with this. Prepare for destroyed (burnt-down level) machine, for datacentre failure, for stolen home server, for scratched blu-ray archives - in short, for the worst. And of course, hope for the best.

Re: GitLab Database Incident – Live Report

#443

Post by neals »

CrLf wrote: post #415
nanch wrote: post #413

Earlier quoted context omitted.

Anybody whose opinion matters understands that this type of event is a process problem, not a person problem. GitLab has always blazed their own trail with their transparency, whether through their open run books, open source code, or in this case their open problem resolution. Kudos to them in whatever manner they want to do it in (with or without names). To be honest, through all of the comments, yours seems the mo…

In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)

knowing exactly how a potential employee handles an error he might have caused? This guy is going to be fighting off job offers, if he hasn't already been.

Re: GitLab Database Incident – Live Report

#444

Post by marcosdumay »

debaserab2 wrote: post #258
iagooar wrote: post #245

Earlier quoted context omitted.

I'm a bit curious here. Do you think that your issues with scalability and reliability have to do with your tech choice (I think it was Ruby on Rails)? Don't want to bash Rails, I'm just genuinely curious, since I come from a Rails background as well and have seen issues similar to yours in the past.

You can check out their codebase here: https://github.com/gitlabhq/gitlabhq Just looking at their gemfile is rather telling: a couple hundred gems. I've always felt that if you're going above 100, you should carefully consider how much your codebase is trying to achieve. They're probably at the point where they really want to think about splitting off of their monolith codebase and into microservices.

> I've always felt that if you're going above 100, you should carefully consider how much your codebase is trying to achieve.

This is a mindset issue. Some communities reject NIH so strongly that you get the opposite problem that everything depends on hundreds of different developers. Gitlab can start some library forks with more stuff integrated, or change communities. Microservices is something that can't help, as all the dependencies will stay just where they are (Gitlab is already uncoupled to some extent).

But, anyway, most of those are stable¹, and I doubt many of Gitlab problems come from dependencies.

1 - They are unbelievably stable for somebody coming from the Python world. When I first installed Gitlab, I couldn't believe on how easy it was to get a compatible set of versions.

Re: GitLab Database Incident – Live Report

#445

Post by louiz »

riebschlager wrote: post #221

I'm another happy GitLab user, but things like this always kinda freak me out. Do any of you use any repo-mirroring strategy? Something a little more automated than pushing to and maintaining separate remotes? For example, would it be worth it to spin up a self-hosted GitLab instance and then script nightly pulls from GitLab.com? Edit: Answered my own question! If anyone else was curious: http://stackoverflow.com/que…

Why would you use gitlab.com if you have a self-hosted Gitlab instance?

Re: GitLab Database Incident – Live Report

#446

Post by msimpson »

Mistakes will happen, sure. But given the following:

> The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented

> Our backups to S3 apparently don’t work either: the bucket is empty

It seems like a lot of their backup and restore procedures were never even tested.

Re: GitLab Database Incident – Live Report

#447

Post by marcosdumay »

Perihelion wrote: post #37
m_fam_wa_k wrote: post #21

Earlier quoted context omitted.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

All of my non-production machines have emojis in PS1 somewhere. It sounds ridiculous, but I know that if I see a cheeseburger or a burrito I'm not about to completely mess everything up. Silly terminal = silly data that I can obliterate.

I got the idea of an emoji prompty that is happy or sad depending on the $? value from somebody here a few years ago.

I run it on any dev environment since then.

Re: GitLab Database Incident – Live Report

#448

Post by blowski »

OskarS wrote: post #420
CrLf wrote: post #415

Earlier quoted context omitted.

In a few years the guy doing the `rm -rf` is going to be on a job interview and someone will recall bits of this report. Enough bits to remember the guy, not enough bits to remember that it wasn't his (individual) fault. Transparency doesn't mean publicly throwing people under the bus. I'm not a GitLab customer, I'm relaxed. :)

Honestly, if I were interviewing the guy, that would almost be a bonus! Like, everyone makes mistakes, we're all human, but I can guarantee you that THAT person will never make that particular mistake ever again. And he's going to be 10 times more diligent than the average engineer in making sure there are good backup/restore procedures.

There's a probably apocryphal story like this about a guy forgetting to refuel a plane. The pilot made sure that guy was solely responsible for refuelling his plane in future, because he knew he'd never forget again.

Re: GitLab Database Incident – Live Report

#449

Post by ndh2 »

virtuabhi wrote: post #85
click170 wrote: post #75

Earlier quoted context omitted.

BBC's Horizon has a really good episode about checklists and how they're used to prevent mistakes in hospitals, and how they're being adopted in other environments in light of that success. It's called How To Avoid Mistakes In Surgery for the interested.

Here's the book on this topic- The Checklist Manifesto https://smile.amazon.com/Checklist-Manifesto-How-Things-Righ...

There's the article that this book is based on The Checklist http://www.newyorker.com/magazine/2007/12/10/the-checklist

Re: GitLab Database Incident – Live Report

#450

Post by marcosdumay »

koolba wrote: post #107
mschuster91 wrote: post #54

Earlier quoted context omitted.

> How does it go unnoticed that S3 backups don't work for so long? My uneducated guess (this one hit a friend of mine): expired/revoked AWS credentials combined with a backup script that doesn't exit(1) on failure and just writes the exception trace to stderr.

I bet it writes to a log file. It just doesn't alert anyone on failure so the log just grows and grows daily with the same error.

Or it alerts people, but on the same channel every other piece of infrastructure alerts them, and they have a severe case of false positives.

I've seen that many more times than I've seen the "no alert" option.

Post reply on HN