171–180 of 621 posts
"Our backups to S3 apparently don’t work either: the bucket is empty" 6/6 failed backup procedures. Looks like they are going to be hiring a new sysadmin/devops person...
Like TJ watson said "fire you? I just spent ten million dollars training you."
What is gitlab storing in their database? From what I understand, the repos were untouched by the DB problems, so what is taking up a third of a terabyte of DB space?
Earlier quoted context omitted.
The best system administrator is the one that has learned from their catastrophic fuck up. To that effect, I still have the same job as I did before I ran "yum update" without knowing it attempts to do in place kernel upgrades. Which resulted in a corrupted RedHat installation on a server we could not turn off.
There is learning from a catastrophic fuck up, and then there is incompetence. Backups is like Day 1, SysAdmin 101. I can't quite grasp how so many different backup systems were left unchecked. Every morning I receive messages saying everything is fine, yet I still go into the backup systems, to make sure they actually did run. In case there was issue with the system alerting me.
We all start at incompetence, but eventually we — wait for it — learn from our experiences. Would you believe that Caesar, Michael Jordan and Steve Wozniak once were so incompetent that they couldn't even control their bowels or tie their shoes? They learned.
Is it possible that the guys in the team running GitLab's operations were misplaced? Certainly — that's a management issue. And I can guar-an-tee you that GitLab now has a team of ops guys who viscerally understand the need for good backups: they'd be insane to disperse that team to the winds.
23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.
23:00-ish YP thinks that perhaps pg_basebackup is being super pedantic about there being an empty data directory, decides to remove the directory. After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Good lesson on the risks of working on a live production system late at night when you're tired and/or frustrated.
Also, as a safety net, sometimes you don't need to run `rm -rf` (a command which should always be prefaced with 5 minutes of contemplation on a production system). In this case, `rmdir` would have been much safer, as it errors on non-empty directories.
Unfortunately, the answer most places is that the diagnostic account (as opposed to the corrective action account) is fully privileged (or worse, root).
Earlier quoted context omitted.
There is learning from a catastrophic fuck up, and then there is incompetence. Backups is like Day 1, SysAdmin 101. I can't quite grasp how so many different backup systems were left unchecked. Every morning I receive messages saying everything is fine, yet I still go into the backup systems, to make sure they actually did run. In case there was issue with the system alerting me.
> There is learning from a catastrophic fuck up, and then there is incompetence. We all start at incompetence, but eventually we — wait for it — learn from our experiences. Would you believe that Caesar, Michael Jordan and Steve Wozniak once were so incompetent that they couldn't even control their bowels or tie their shoes? They learned. Is it possible that the guys in the team running GitLab's operations were mispl…
Earlier quoted context omitted.
Or even instead of any kind of rm command. mv is less subtle. I tend to prefer `mv x $(date +%Y%m%d_%s_)x` where: %Y - 4 digit year %m - 2 digit month %d - 2 digit day _ - underscore literal %s - linux timestamp (seconds since epoch) This ensures that the versions you're 'removing' will be lexically sorted from newest to oldest in a way that is easy to interpret and also works if you need to try more than once in a d…
This needs to be the first thing anyone who works with stateful systems learns. NEVER rm. mv is insufficient. mv dir dir.bak.`date +%s` has prevented data loss for me several times.
At some level you have to run an rm, and you better hope you do it right in the middle of an emergency with people breathing over your shoulder.
In an ideal world, this wouldn't ever happen, but it does. Inherited/legacy systems suck.
As a side note, I just checked our S3 Gitlab backup bucket and it does have backups for every day for the last year (1.8 GB each yikes) so instead of failing to create the backups, its actually failing to delete the older ones! :)
>1. LVM snapshots are by default only taken once every 24 hours. YP happened to run one manually about 6 hours prior to the outage >2. Regular backups seem to also only be taken once per 24 hours, though YP has not yet been able to figure out where they are stored. According to JN these don’t appear to be working, producing files only a few bytes in size. >3. SH: It looks like pg_dump may be failing because PostgreSQ…