51–60 of 176 posts
That sucks. Everyone has been there. On my first sysadmin job I did an rm /* as root and then discovered my backups across the network to a remote tape unit had a buffering issue and were basically useless.
I know that this is the time to say "you should have tested your backups", but I never tested mine. How does one test a (personal) backup strategy?
Earlier quoted context omitted.
> manual restore testing They didn't test restores of the actual data being backed up. They did a manual restore test of data that they manually backed up.
I want to feel bad for them, and I do, but really I’m not sure what you expect. They never tested their backup. They tested their ability to manually back up, but I’m not sure why they would ever think to conflate that with automatic processes. If they had automatic backups, why were they relying on manual ones for testing?
They tested their ability to do a backup, but never went through with actually backing it up - automating it - let alone making it a process to do before you make changes in “production”. Amateur. I will give this credit though. It’s hard. Not everyone can do it. Running production that is. Anyone can build for production but actually keeping it live, running, and working nominally with proper SLA and DR? It’s hard. I’m curious what the volumes were even for? I tend to not use storage outside of blob, databases, queues.
> While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data. Can someone explain this? How did they test restores, if the actual restore failed to come up with…
Velero can be configured to run on a schedule, but the scheduled command was apparently not the exact same command they were using to perform the manual tests - the scheduled job was missing part of the command, basically.
> While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data. Can someone explain this? How did they test restores, if the actual restore failed to come up with…
Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.
...does absence signify deletion?
Some projects decide yes, if the YAML specifying your resource is absent on the next reconciliation, then delete the corresponding K8s resources.
Then some projects go with "no, deletion must be explicit, set this flag in your YAML".
But the last approach doesn't work as nicely with GitOps, as you have to do a two stage workflow to delete - first set the delete flag, then once the resources are deleted, then delete the YAML.
But I'm okay with deletion being harder to do if it makes it harder to well, accidentally delete all your stuff.
Because I've never met a K8s workflow where resources were commonly deleted, usually it's creation and update 90% of the time.
But I can understand it'd be annoying if you ended up with dangling resources if you missed an explicit delete flag.
> While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data. Can someone explain this? How did they test restores, if the actual restore failed to come up with…
They were running for years on the cusp of total failure and had automated restoration tests that caused a false sense of security in the tooling.
The second thing I did was adjust the restoration tooling to validate data existence and over time added validation tests (percent of data matched current live systems, specific fields and values were there, etc).
It's just too easy to screw up, doubly so when time constrained and alone doing the best you can without any oversight.
Earlier quoted context omitted.
This is a dumb question, but why can't people just make things simpler? Tools are supposed to make work easier and make things more efficient, but this sort of complexity just seems to hurt, doesn't it?
I wonder if it has to do with mismatches between -as-cattle and -as-pets philosophies and usecases. From the post, it looks like they were using k8s. It's hard to tell given the, you know, total data loss, but it looks like the instance had at max triple-digit users. In my experience running small community stuff like this, the -as-cattle tooling, even though I'm familiar with it from work, is way overkill and introd…
You could just run docker or podman or whatever alone on a VM and start/stop containers entirely manually.
That sucks. Everyone has been there. On my first sysadmin job I did an rm /* as root and then discovered my backups across the network to a remote tape unit had a buffering issue and were basically useless.
I know that this is the time to say "you should have tested your backups", but I never tested mine. How does one test a (personal) backup strategy?
And when you're setting up that test, get the password and any other details to access the backups via a back up method too.
> While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data. Can someone explain this? How did they test restores, if the actual restore failed to come up with…
When I started at a large company a few years back, the company specified restoration test was just that the archive restored successfully onto a server, not that there was anything actually in the archive. Digging into it, the archives were all empty due to a commit a few years previous that added an incorrect exclude option that ended up excluding all files. They were running for years on the cusp of total failure…