Total data loss after botched GitOps and failed backups
firefish.social
Total data loss after botched GitOps and failed backups
1–10 of 176 posts
Re: Total data loss after botched GitOps and failed backups
#2It's odd to me when people say backups are a feature.
They are not.
Restoration, however, is.
And has to be tested...
Re: Total data loss after botched GitOps and failed backups
#3The more layers of automation we add, the more invisible points of failure. Magic is great until it isn’t.
I feel their pain. The sinking feeling when you realize that data is gone and not coming back is an awful experience.
Re: Total data loss after botched GitOps and failed backups
#4Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow.
I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.
Re: Total data loss after botched GitOps and failed backups
#5I don't understand the backup failure, how did you not notice your PVCs weren't being backed up? Like wouldn't you realize they were too small or happening too fast? It seems like the root cause is people not having their work checked from first principles.
Re: Total data loss after botched GitOps and failed backups
#6Oof. I've been there. The rush of adrenaline and then just feeling completely wrecked and empty is something that sticks with you for a long time. Take care, lone admin - don't be too hard on yourself.
Re: Total data loss after botched GitOps and failed backups
#7Anyone who would feel a "crippling loss" if their data disappeared must execute a real world real failure test at least annually. Disconnect the storage, attach new storage, and go.
Re: Total data loss after botched GitOps and failed backups
#8Is it all fediverse data? If so, could the data be reconstructed from copies scattered amongst caches on other parties' servers?
Re: Total data loss after botched GitOps and failed backups
#9Automation is wonderful for creating things, configuring things and moving things.
Automation should never "clean up". Do your cleanup manually, or this is what you get.
Re: Total data loss after botched GitOps and failed backups
#10It's odd to me when people say backups are a feature. They are not. Restoration, however, is. And has to be tested...
It sounds like they did test restorations, but didn't correctly verify the results. It's not very clear though.
>While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually.