> So in other words, out of 5 backup/replication techniques deployed, none are working reliably or set up in the first place.
'Check Your Backups Work' Day
61–70 of 152 posts
Re: 'Check Your Backups Work' Day
#62Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…
I have nothing, but respect for the GitLab team.... offering live notes for the recovery is stunning.
Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers.
Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it damages it !
Re: 'Check Your Backups Work' Day
#63Earlier quoted context omitted.
Heh, "Don't be an April Fool" reminded me of the old annual "Internet Spring Cleaning" [0] ritual. [0]: http://www.snopes.com/holidays/aprilfools/cleaning.asp
I wonder if anyone would fall for it now...
Along the same lines as https://xkcd.com/1053/, I believe so.
Re: 'Check Your Backups Work' Day
#64> After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Everyone is talking about backups, but why not about this? How is it even possible to delete the production database by accident? Why does he have SSH access there? Why do they test their database replication in production? Why are they fire-fighting by changing random database settings _in production_? I know th…
How would you resolve a production data replication issue in an environment other than production, or without access to the system?
Re: 'Check Your Backups Work' Day
#65When stuff "just works", you don't need to check your backups. I fully trust my iPhone's iCloud backups, my Time Machine backups, and my cloud rsyncs. Time Machine also lets me know if they get corrupted, or if I haven't backed up in a while. That's how backups should work - an adage of "you don't have backups unless you check them" just won't work for most people.
You might want to check your Time Machine backups by hand from time to time using the tmutil[1] tool.
My Time Machine backups in the past have been missing gigabytes of data, without me telling anything about it. And not just volatile data, like temp files or caches, but photos, music and documents.
[1]: http://osxdaily.com/2012/01/21/compare-time-machine-backups-...
Re: 'Check Your Backups Work' Day
#66Can't someone working for the company I bank with accidentally delete my debt and its backups? :P
Re: 'Check Your Backups Work' Day
#67Earlier quoted context omitted.
I have nothing, but respect for the GitLab team.... offering live notes for the recovery is stunning.
Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it…
Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and no written plan). The solutions above are just guesses at this point, we'll dive into this in the coming days and will communicate what we will do in a blog post.
Re: 'Check Your Backups Work' Day
#68Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…
I have nothing, but respect for the GitLab team.... offering live notes for the recovery is stunning.
Completely agree. This is 1000x better than the typical cooperate PR response that says nothing which we'll usually get. Kudos for their honesty and openness, even in a somewhat embarrassing situation such as that.
Re: 'Check Your Backups Work' Day
#69Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…
Is it too much to ask for a few days dedicated to this? I mean, pretent that your critical infastructure goes up in smoke - can you recover, and how quickly? For most companies it simply means that you try to restore the machines from your backup. You can do this as everything else continues as normal.
Re: 'Check Your Backups Work' Day
#70> After a second or two he notices he ran it on db1.cluster.gitlab.com, instead of db2.cluster.gitlab.com Everyone is talking about backups, but why not about this? How is it even possible to delete the production database by accident? Why does he have SSH access there? Why do they test their database replication in production? Why are they fire-fighting by changing random database settings _in production_? I know th…
I would start by giving them more descriptive names... calling it db1 and db2 is a sure way to trip someone some day.