I noticed the issue when I was pushing code earlier today.
Hopefully this gets resolved soon.
You guys are doing a great job. Keep up the good work!
GitLab Database Incident – Live Report
31–40 of 621 posts
Re: GitLab Database Incident – Live Report
#32Earlier quoted context omitted.
Good lesson on making command prompts on machines always tell you exactly what machine you're working on.
I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
I do this too, but in this case both machines were production, so this alone would not have sufficed. The system-default prompts on the other hand are universally garbage.
Re: GitLab Database Incident – Live Report
#33Re: GitLab Database Incident – Live Report
#34Earlier quoted context omitted.
Good lesson on making command prompts on machines always tell you exactly what machine you're working on.
I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
Yep, good idea. The same thing has been suggested by team members http://imgur.com/a/TPt7O
Re: GitLab Database Incident – Live Report
#35This is the stuff my nightmares consist of after 900 consecutive days of being on call (and counting).
Are you a one man team or...? My wife would probably leave me if I was on-call for that long.
Re: GitLab Database Incident – Live Report
#36"So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."
[deleted]
Re: GitLab Database Incident – Live Report
#37Earlier quoted context omitted.
Good lesson on making command prompts on machines always tell you exactly what machine you're working on.
I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
All of my non-production machines have emojis in PS1 somewhere. It sounds ridiculous, but I know that if I see a cheeseburger or a burrito I'm not about to completely mess everything up. Silly terminal = silly data that I can obliterate.
Re: GitLab Database Incident – Live Report
#3823:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.
Also, as a safety net, sometimes you don't need to run `rm -rf` (a command which should always be prefaced with 5 minutes of contemplation on a production system). In this case, `rmdir` would have been much safer, as it errors on non-empty directories.
These days, I've been very implicit in how I run rm. To the extent that I don't do rm -rf or rmdir (edit: immediately), but in separate lines as something like:
pushd dir ; find . -type f -ls | less ; find . -type f -exec rm '{}' \; ; popd ; rm -rf dir
Re: GitLab Database Incident – Live Report
#39Earlier quoted context omitted.
I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
How do you go about colour coding your terminal?
I assume he color coded the prompt. You can use ANSI color escape codes in there to e.g. color your hostname.
Here's a generator for Bash: http://bashrcgenerator.com/, the prompt's format string is stored in the $PS1 variable.
Re: GitLab Database Incident – Live Report
#40"So in other words, out of 5 backup/replication techniques deployed none are working reliably or set up in the first place."
Does this mean whatever was in that database is gone, with no available backups?
Is this an SOA where important data might lie in another service or data store, or is this a monolithic app and DB that is responsible for many (or all) things?
What was stored in that database? Does this affect user data? Code?