If this was with Argo shouldn’t finalizes prevented this?
Total data loss after botched GitOps and failed backups
171–176 of 176 posts
Re: Total data loss after botched GitOps and failed backups
#172> While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data. Can someone explain this? How did they test restores, if the actual restore failed to come up with…
When I started at a large company a few years back, the company specified restoration test was just that the archive restored successfully onto a server, not that there was anything actually in the archive. Digging into it, the archives were all empty due to a commit a few years previous that added an incorrect exclude option that ended up excluding all files. They were running for years on the cusp of total failure…
Empty backups or exponentially I creasing in size backups of backups. The yin and yang of backup bugs.
Re: Total data loss after botched GitOps and failed backups
#173Earlier quoted context omitted.
I think this pattern is fine most of the time. I hate it for PVCs. At the least with cloud-like volumes it should leave a final snapshot that sticks around for a time. I have all the sympathy for the author. I’m sure they don’t feel good right now, but I hope they continue contributing. They’ve learned a hard lesson, but putting it back into their practice is the only way to make it count.
You can set a PVC's reclaim policy to "Retain" (default is "Delete"). When the namespaced claim is deleted, the volume will still be around until it is manually deleted. The docs even suggest to do this for "precious data".
Why is catastrophically unsafe behavior the default? This is Ford Pinto-level engineering on the part of the k8s team.
Re: Total data loss after botched GitOps and failed backups
#174Here we go again, a lot of comments saying the giant ball of knives is too dangerous.
I work with the giant ball of knives every day, wrapped in my custom-tailored full-body kevlar bubble, and I barely ever have any catastrophic life-threatening accidents!
The giant ball of knives is not the issue. Your carelessness when working around the giant ball of knives is the issue. Look inward.
Re: Total data loss after botched GitOps and failed backups
#175Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.
xkcd: automate, from auto meaning self, and mate meaning...
Mate means death in te reo Māori, which also kinda works here.
Re: Total data loss after botched GitOps and failed backups
#176Earlier quoted context omitted.
You can set a PVC's reclaim policy to "Retain" (default is "Delete"). When the namespaced claim is deleted, the volume will still be around until it is manually deleted. The docs even suggest to do this for "precious data".
Why is catastrophically unsafe behavior the default? This is Ford Pinto-level engineering on the part of the k8s team.
Guess it's the ol 'cattle not pets' taken to the extreme.