'Check Your Backups Work' Day
71–80 of 152 posts
Re: 'Check Your Backups Work' Day
#72Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…
This page did a lot to raise my awareness of my own inadequate backup situation. Any hurt feeling of the CEO caused by this page will be outweighed by the greater good of the message being spread.
Re: 'Check Your Backups Work' Day
#73Earlier quoted context omitted.
You know, I remember reading years ago that during the launch preparation for Apollo 16(?), they did a routine pressure test of the Command Module atop the rocket a few days before launch. The technician in charge made the simple error of forgetting to open a pressure release valve during the test which lead to an overpressurisation of the CM causing significant damage, including separation of the heat shield. The EN…
>When the Apollo launchpad manager was asked if he had fired the technician in question, then answer was allegedly "Nope. He is the one guy on the next launch team that I know will NEVER make the same mistake again." This same general story appears in different forms with million dollar trading errors, etc. But I have to wonder if it's really that good of a lesson. If you truly take it to heart, you could potentially…
Make one mistake, it's an expensive training lesson.
Make the same mistake again, you get fired.
Re: 'Check Your Backups Work' Day
#74Imagine if you're the CEO of Gitlab and seeing this right now. I think showing compassion and solidarity may be a better response. The problem of restoring non-existing backups should be treated as a more serious problem in our industry. This happens too often and not because people who made them were careless people but because to catch any errors when the backups are not working can take unreasonable amount of time…
I have nothing, but respect for the GitLab team.... offering live notes for the recovery is stunning.
it is also very, very risky from a liability perspective, both for insurance purposes and third-party indemnification. That's why most companies will keep as silent as possible.
Re: 'Check Your Backups Work' Day
#75In my day we used to develop "Disaster Recovery" programs. They were massive, and we tested on a regular basis including renting massive systems from IBM and flying the team to the IBM data center to run a full restore of everything. End business users had to login afterwards and sign off.
I understand we "live in a different world" is the favorite motto these days. But do we really? If anything data is bigger, more complex, not in one place, you can't just ship a truck load of tapes and three people somewhere to test.
IMHO the more we try to reinvent technology the more we realize some of the things we all felt were weighing us down were actually smart ideas brought on not by fear but by real life experience.
And the pendulum will swing once more here and back again at some point in the future.
Re: 'Check Your Backups Work' Day
#76Gitlabs behavior is a testament to success ! No hidding, no eupheminization, their live doc stream actually made me question what things did I do on my systems. Looks like convergent evolution in some parts, like prompt changes. Thanks Gitlab.
From someone who works at GitLab, thank you for your kind words! Our infrastructure team is working hard!
Re: 'Check Your Backups Work' Day
#77Not referring to Gitlab's incident but the general problem with this stuff is too often people treat something like "make a backup" too literally.
As in, they think the act of generating a backup file is the last stage of the process and they are done with it. Maybe they go the extra mile and throw it in a cronfile too.
What you have to do is to consider any and all backups non-existent until you have a complete backup strategy.
In other words you have to appreciate that "having a backup" is a means to an end and another way of saying "being able to successfully recover from data loss".
So just generating a file is not sufficient.
You have to complete a successful test restore before you can call it a backup.
You have to have heart-beat measures in place to make sure you will not be impacted by a silent failure (example: check $last_successful_backup >= last X hours).
You have to periodically manually check that your automated checks work ("simulate" backups not being generated and wait for alert).
Far too often people don't appreciate the depth and the weight that a phrase like "backup plan" carries.
So you may ask them to "take care of backups" they will go run pg_dump and mysqldump and say "It's done". No goddamn it, it's not done.
Re: 'Check Your Backups Work' Day
#78In my day we used to develop "Disaster Recovery" programs. They were massive, and we tested on a regular basis including renting massive systems from IBM and flying the team to the IBM data center to run a full restore of everything. End business users had to login afterwards and sign off. I understand we "live in a different world" is the favorite motto these days. But do we really? If anything data is bigger, more…
I don't recognise the 'different world' you describe. Practiced disciplined DR is a key part of modern software engineering at large and small companies across the startup/enterprise spectrum (in my experience). The popularity of tools like Netflix's Chaos Monkey/Gorilla/Kong server as testament to that.
That's not to say all companies do it (it seems GitLab didn't) but the tone of your comment doesn't reflect a lot of people's experiences.
Re: 'Check Your Backups Work' Day
#79Earlier quoted context omitted.
Oh wow, some interesting stuff in here. Looks like they use MySQL as a queue for scheduling the ORC Peons? Would have loved to hear more about why they did that.
Because it works perfectly fine? I tend to write anyone off who scoffs at simple database-as-queue designs without understanding what the scaling requirements are. You can use a database as a job queue for 10s of thousands of jobs per day without any sweat.
Scoffing? GP said it was "interesting stuff" and that they'd "love to hear more". What's wrong with that?
Re: 'Check Your Backups Work' Day
#80Earlier quoted context omitted.
Agreed ! Even when they screw things up, they make people benefit from it! Kudos for such a good spirit and dedication to your customers. Everybody screw up from time to time in our industry. And when this happens, you have 2 types of guy : those who try to hide it, and those like Giltlab team who communicate as fast as they can because they respect their customers. Paradoxically, to me, it creates more trust than it…
Thanks for the kind words. I'm sorry for letting our users down. We'll ask the the 5 why's https://en.wikipedia.org/wiki/5_Whys We need to go from the initial mistake (wrong machine, solve by better hostname display and colors), to the second (not having a recent backup), to the third (not testing backups), to the fourth (not having a script for backup restores), to the fifth (nobody in charge of data durability and…
make sure everyone is doing ok, don't let them beat themselves over it
all the love and support in the world towards the team