I like to always try to automate stuff as much as possible to remove human error. It's easy to forget an arg or other parameter on a script or even know what the arg was for in the first place. Sounds like their last backup was 24 hrs ago. Having backups are like having good security, you don't realize how important they are until it's too late. Reminds me of this old meme: http://www2.rafaelguimaraes.net/wp-content/…
GitLab Database Incident – Live Report
341–350 of 621 posts
Re: GitLab Database Incident – Live Report
#342> Work is interrupted due to this, and due to spam/high load on GitLab.com
This is why spam should be illegal. The advertisers, the ISPs harboring them or their country should be taken on for damages. This not only prevented Gitlab from doing business, but also people who depend on them from doing business. It's criminal.
Re: GitLab Database Incident – Live Report
#343Earlier quoted context omitted.
Or even instead of any kind of rm command. mv is less subtle. I tend to prefer `mv x $(date +%Y%m%d_%s_)x` where: %Y - 4 digit year %m - 2 digit month %d - 2 digit day _ - underscore literal %s - linux timestamp (seconds since epoch) This ensures that the versions you're 'removing' will be lexically sorted from newest to oldest in a way that is easy to interpret and also works if you need to try more than once in a d…
This needs to be the first thing anyone who works with stateful systems learns. NEVER rm. mv is insufficient. mv dir dir.bak.`date +%s` has prevented data loss for me several times.
To this I would add: always pass "-i" to any interactive use of mv. This has also helped me prevent data loss a few times.
Re: GitLab Database Incident – Live Report
#344Earlier quoted context omitted.
On a production server this delicate, (eg: production db), I'd replace /bin/rm with a script that moves stuff to /trash. Then, always remember to delete /trash a few days later.
Maybe add a bash function to check the path and ask the magic question: "Do you really want to delete the dir XYZ in root@domain.com?" .... but then again when you're in panic mode you might either misread the host or hit 'y' without really reading what's in front of you. The best thing to do is to never operate with 2 terminals simultaneously, when one of them is a production env, better login/logout or at least min…
At my last place, the terminal on prod servers had a red background, HA servers was amber and staging servers green - worked a treat.
Re: GitLab Database Incident – Live Report
#345Earlier quoted context omitted.
Two points: 1) Patio11 touches on a very good lesson, in passing, in an article about Japanese business[1]: While raw programming ability might not be highly valued at many Japanese companies, and engineers are often not in positions of authority, there is nonetheless a commitment to excellence in the practice of engineering. I am an enormously better engineer for having had three years to learn under the more senior…
I just wish big companies were more willing to remove this scar tissue.
Or even not scarify themselves before potential Big Bad happens.
Pushing the metaphor a bit too far.
Re: GitLab Database Incident – Live Report
#34623:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.
Just a thought, but is it possible to override destructive commands to confirm the hostname by typing it before running the command?
Re: GitLab Database Incident – Live Report
#347I know it's not a laughing matter (and suddenly my working day just seems much easier), but when I read
"YP says it’s best for him not to run anything with sudo any more today"
I couldn't but smile. Yeah, well, that's a good point probably : )
Re: GitLab Database Incident – Live Report
#348Earlier quoted context omitted.
I like to color code my terminal. Production systems are always red. Dev are blue/green. Staging is yellow.
I use iterm2's "badging" to set a large text badge on the terminal of the name of the system as part of my SSH-into-ec2-systems alias: i2-badge () { printf "\e]1337;SetBadgeFormat=%s\a" $(echo -n "$1" | base64) } It's not quite as good as having a separate terminal theme, but then I haven't been able to use that feature properly. :(
This is a great idea! Documentation https://www.iterm2.com/documentation-badges.html -- supports emojis too fwiw
Re: GitLab Database Incident – Live Report
#349Earlier quoted context omitted.
I think it's a staff member. Can't remember first name, Yuri maybe, who is fairly active with the project.
Nope, that would be me.
Dude, we've all been there. You are neither the first nor the last. It's never a single person. One day the technical side of our little startup collectively went to a conference (back then it was only two engineers plus a technical leader) while our server DoS'd itself by broken mail processing... we had a rough night in a Belgium hotel figuring out who attacked the site to realize it was ourselves. The 10k block of missing image IDs always stood as a reference not to leave even a low traffic site unmonitored. It happens.
Re: GitLab Database Incident – Live Report
#350"The replication procedure is super fragile, prone to error, relies on a handful of random shell scripts, and is badly documented"
This is true of many databases, but in my experience is particularly true of postgres. It's a marvelous single-instance product, but I've never really found any replication/ha tech for it that I've been that happy with. I've always been a bit nervous about postgres-backed products for this very reason.
I'd be interested what other people's take on this is, though.