Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Total data loss after botched GitOps and failed backups

firefish.social

Post reply on HN
Print view View original

21–30 of 176 posts

Re: Total data loss after botched GitOps and failed backups

#21

Post by belltaco »

bg24 wrote: post #11

Very sad. But also a miss on admin's part, even if unfortunate. They did not realize during manual restore testing that volume was not being backed up. "Yes and also apparently no. We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim dat…

> manual restore testing

They didn't test restores of the actual data being backed up. They did a manual restore test of data that they manually backed up.

Re: Total data loss after botched GitOps and failed backups

#22

Post by noobermin »

macintux wrote: post #3

The more layers of automation we add, the more invisible points of failure. Magic is great until it isn’t. I feel their pain. The sinking feeling when you realize that data is gone and not coming back is an awful experience.

This is a dumb question, but why can't people just make things simpler? Tools are supposed to make work easier and make things more efficient, but this sort of complexity just seems to hurt, doesn't it?

Re: Total data loss after botched GitOps and failed backups

#24

Post by [deleted] »

simula67 wrote: post #20

> While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data. Can someone explain this? How did they test restores, if the actual restore failed to come up with…

[deleted]

Re: Total data loss after botched GitOps and failed backups

#25

Post by defaultcompany »

pdntspa wrote: post #4

Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.

Ugh I did something like this once when I wrote a script to rsync one volume to a backup volume. Everything was great until someone accidentally rm -r ‘d the source volume and the script happily deleted everything on the backup volume to keep it in sync.

Re: Total data loss after botched GitOps and failed backups

#27

Post by nicbou »

pnw wrote: post #26

That sucks. Everyone has been there. On my first sysadmin job I did an rm /* as root and then discovered my backups across the network to a remote tape unit had a buffering issue and were basically useless.

I know that this is the time to say "you should have tested your backups", but I never tested mine.

How does one test a (personal) backup strategy?

Re: Total data loss after botched GitOps and failed backups

#28

Post by livueta »

noobermin wrote: post #22
macintux wrote: post #3

The more layers of automation we add, the more invisible points of failure. Magic is great until it isn’t. I feel their pain. The sinking feeling when you realize that data is gone and not coming back is an awful experience.

This is a dumb question, but why can't people just make things simpler? Tools are supposed to make work easier and make things more efficient, but this sort of complexity just seems to hurt, doesn't it?

I wonder if it has to do with mismatches between -as-cattle and -as-pets philosophies and usecases. From the post, it looks like they were using k8s. It's hard to tell given the, you know, total data loss, but it looks like the instance had at max triple-digit users. In my experience running small community stuff like this, the -as-cattle tooling, even though I'm familiar with it from work, is way overkill and introduces far more brittleness than it prevents. My go-to for stuff like this is -as-pets: gimme a box, a daemon, some rc scripts and maybe a backup client. Admin complexity and burnout are huge problems for volunteer sysadmins, and IMO the burden of -as-cattle tooling unnecessarily exacerbates both.

I've had an irc daemon running on the same metal for almost 15 years now, uninterrupted besides occasional OS upgrades and patches. There is basically nothing else happening on the host. It is dumb simple and just works.

Some of that may be on devs, actually: distribution-as-docker-container at the expense of real packaging seems to be on the rise.

Re: Total data loss after botched GitOps and failed backups

#29

Post by LoganDark »

nicbou wrote: post #27
pnw wrote: post #26

That sucks. Everyone has been there. On my first sysadmin job I did an rm /* as root and then discovered my backups across the network to a remote tape unit had a buffering issue and were basically useless.

I know that this is the time to say "you should have tested your backups", but I never tested mine. How does one test a (personal) backup strategy?

> How does one test a (personal) backup strategy?

Boot into one. (Clone it onto a spare drive and use that as a boot target.)

If it works flawlessly, there's your backup. If it doesn't, fix your backups so they're bootable.

Re: Total data loss after botched GitOps and failed backups

#30

Post by oneepic »

>I won’t personally be bringing back outdoors.lgbt or firefish.lgbt. Being an admin has been one of the most fulfilling things I have done in a long time and you all have made it such an amazing experience., however, I need to take a step back. I would love to hand the domains over to someone with a similar passion for creating a safe and welcoming community.

I hope the author isn't only quitting because of a feeling of guilt or shame. If that's true, it's certainly ok to give it another try. Tech is toxic enough depending on where you look, and that's not even counting the self-hurt that some of us do to ourselves.

Post reply on HN