101–110 of 152 posts
Earlier quoted context omitted.
The point is that you could have a bug in the verification process that returns "all good" when it really isn't.
I have a SELECT of the most recent updated_at fields for a few tables emailed to me so I can eyeball the values
Earlier quoted context omitted.
I have nothing, but respect for the GitLab team.... offering live notes for the recovery is stunning.
it is also very, very risky from a liability perspective, both for insurance purposes and third-party indemnification. That's why most companies will keep as silent as possible.
And presumably even if your notes are kept private, it'll still come out in discovery if you end up in court?
What are the risks here? I'd be interested to learn more.
Earlier quoted context omitted.
Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it…
Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and…
I've realized in the past that in day-to-day situations it is more likely to lose data because of temporary programmer carelessness. Examples: deleting the backup version of a folder, deleting the copy on the wrong server, etc. This seems similar to what happened at Gitlab.
How to protect oneself from this failure mode? Can we design a better system than to assume that every human command is well considered? (Sort of like guard rails, when purging backups)
While we're discussing the importance of backups, I would like to pause for a minute to think about a common systemic failure model that simply making backups doesn't solve . I've realized in the past that in day-to-day situations it is more likely to lose data because of temporary programmer carelessness. Examples: deleting the backup version of a folder, deleting the copy on the wrong server, etc. This seems simila…
We had a MongoDB server (with a read-replica) in our production environment (this was an ec2 instance with MongoDB).
One day, a dev accidentally deleted the main collection in the DB during a night coding session. Next morning, when we realized that, we went straight to the daily backups that we have been doing. It happened that for some reason the backup of the previous 2 or 3 days did not work.
We had to get into Mongo-OpLog (which was on only because of the read-replica) and reconstruct the missing 3 days from it.
That was fucking scary.
Look, shit happens... we don't need to make fun of people for it. We all cut corners at times... when we are lucky, nobody notices. When we aren't...
Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…
That's why we need awareness days.
Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…
> even then it could still just stop working one day This is where automated testing helps. My mail server has a sister VM out in the wild that once a day picks up the latest backup from the offsite-online backups and restores it, sending me a copy of the last message received according to the data in that backup. If I don't get the message, restoring the backup failed. If the message looks too old then making or tra…
Now that I'm developing email stuff I get all the time the existential crisis of "how to I make sure my testing and alerting infrastructure is working?"
Really, how can you be sure your restoration tests are running correctly and their messages get all the way to you?
I think I'll add periodic positive messages for my restoration procedures. Stuff like (if success and day % 23 == 0 then email_something). Still doesn't guarantee the test is correct.