Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GitLab Database Incident – Live Report

docs.google.com

Post reply on HN
Print view View original

211–220 of 621 posts

Re: GitLab Database Incident – Live Report

#211

Post by scrame »

Perihelion wrote: post #37
m_fam_wa_k wrote: post #21

Earlier quoted context omitted.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

All of my non-production machines have emojis in PS1 somewhere. It sounds ridiculous, but I know that if I see a cheeseburger or a burrito I'm not about to completely mess everything up. Silly terminal = silly data that I can obliterate.

ha! I do the same thing with figlet and cowsay, it prints a big dragon saying "welcome to shell!", if I see that, then I know I'm on a box I own/have sudo/am me. it's a good visual reminder. I don't fuss with prompts much, but this is a pretty good idea!

Re: GitLab Database Incident – Live Report

#212

Post by namepros »

hobarrera wrote: post #134
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.

This sounds good until you consider that many systems are utilizing multiple drives. When someone is expecting to delete a large file and it ends up on a different drive, problems could arise.

Re: GitLab Database Incident – Live Report

#213

Post by gvb »

hardwaresofton wrote: post #207
gizmo wrote: post #120

Earlier quoted context omitted.

What if I told you all of society is held together by duct tape? If you're surprised that startups cut corners you're in for a rude awakening. I'm frequently amazed anything works at all.

The real question is what holds together duct tape?

The dark side.

Re: GitLab Database Incident – Live Report

#214

Post by ubercow »

gknoy wrote: post #45
m_fam_wa_k wrote: post #21

Earlier quoted context omitted.

I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.

I use iterm2's "badging" to set a large text badge on the terminal of the name of the system as part of my SSH-into-ec2-systems alias: i2-badge () { printf "\e]1337;SetBadgeFormat=%s\a" $(echo -n "$1" | base64) } It's not quite as good as having a separate terminal theme, but then I haven't been able to use that feature properly. :(

Didn't know about this feature, gonna have to use this thanks!

Re: GitLab Database Incident – Live Report

#215

Post by jamesmiller5 »

CaliforniaKarl wrote: post #160

First off, my heartfelt commiserations for the GitLab team here. My suggestion: Start watching an hour-long video; the rsync will finish right when it gets to the good part! I wonder if a future project might be to have the DB-stored stuff use Git as a replication back-end. Like, for example, having each issue be a directory, and individual comments be JSON files. It would never (normally) be the data store "of recor…

Gerrit is working on just this. It uses `refs/meta/config` branch for project configuration and is moving its database dependencies into a git database. Reviews are stored in refs/changes/* . Backing-up a project & verifying it's integrity is simple as `git clone --mirror`

Re: GitLab Database Incident – Live Report

#216

Post by nihonde »

ryandrake wrote: post #185
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

If you get the chance to observe pilots operating in the cockpit, I'd recommend it. Every important procedure (even though the pilot has it memorized) is done with a checklist. Important actions are verbally announced and confirmed: "You have the controls" "I have the controls". Much of flight training deals with situational awareness and eliminating distractions in the cockpit. Crew Resource Management[1]. 1: https:…

This reminds me of Japanese train crews and factory workers who use hand signals and audible call-outs to reinforce checklists with muscle memory.

Re: GitLab Database Incident – Live Report

#217

Post by late2part »

schleppy_oc wrote: post #208
gizmo wrote: post #19

This is painful to read. It's easy to say that they they should have tested their backups better, and so on, but there is another lesson here, one that's far more important and easily missed. When doing something really critical (such as playing with the master database late at night) ALWAYS work with a checklist. Write down WHAT you are going to do, and if possible, talk to a coworker about it so you can vocalize th…

The best Ops people I have worked with (looking at you Dennis and Alan) repeat everything back that I say. More than once I have caught mistakes in my approach simply by hearing someone else repeat back exactly what I just said.

The best ops people have made all the mistakes they're helping you avoid.

Re: GitLab Database Incident – Live Report

#218

Post by mrbill »

imcoconut wrote: post #198
verytrivial wrote: post #140

Earlier quoted context omitted.

I saw that Space Shuttle landing video that was kicking around recently. In that they also had explicit "I agree" responses to any observation like "You're a bit below flight path". Quick, positive acknowledgment of anomalous events or deviations. Seemed really ... sane.

do you happen to have the link for that video (and related discussion if posted on HN)?

Maybe https://www.youtube.com/watch?v=8W4cfIyNvts ?

Re: GitLab Database Incident – Live Report

#219

Post by 1_2__3 »

ams6110 wrote: post #9

23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.

I think I can count on one hand the number of times I've run an rm command on a production server. I'll move it at worst, and only delete anything if I'm critically low on disk space. But even then I don't even like typing those characters if I can avoid it, regardless of if I'm running as root or a normal user.

Re: GitLab Database Incident – Live Report

#220

Post by jlgaddis »

Apparently the BOFH works for Gitlab these days:

> It's backup day today so I'm pissed off. Being the BOFH, however, does have it's advantages. I reassign null to be the tape device - it's so much more economical on my time as I don't have to keep getting up to change tapes every 5 minutes. And it speeds up backups too, so it can't be all bad can it? Of course not. --bofh, episode #1

Post reply on HN