Total data loss after botched GitOps and failed backups • Hacker News Board

Re: Total data loss after botched GitOps and failed backups

#31

Post by hnbear » Mon, Aug 28, 2023, 4:19 AM UTC

nicbou wrote: post #27

pnw wrote: post #26

That sucks. Everyone has been there. On my first sysadmin job I did an rm /* as root and then discovered my backups across the network to a remote tape unit had a buffering issue and were basically useless.

I know that this is the time to say "you should have tested your backups", but I never tested mine. How does one test a (personal) backup strategy?

Personally my personal files like important docs go to:

- iCloud (automatically)

- Local nightly rsync to a zfs share from local Docs folder - Backblaze

- Some crit stuff to OneDrive

- TimeMachine on a local Linux server at home

Periodically I check them all, verify file counts and sizes, spot check files. Periodically isn’t that often, but I’ve actually needed backed up files and used it.

Data volume are low though.

I keep Photos libraries similarly backed up. That’s harder to verify, although sizes is pretty good. Even if the library metadata gets fried the raw files still exist in the folder structure.

Re: Total data loss after botched GitOps and failed backups

#32

Post by lamontcg » Mon, Aug 28, 2023, 4:20 AM UTC

This all seems way overly complicated for what could probably be a few services running on a VM or four. Why does Kubernetes/Argo/Helm need to get involved? Why couldn't the whole architecture diagram look a lot more like HN? I feel like we've entirely lost our way with complexity.

Re: Total data loss after botched GitOps and failed backups

#33

Post by jakelazaroff » Mon, Aug 28, 2023, 4:20 AM UTC

sp332 wrote: post #18

devmor wrote: post #9

Automation is wonderful for creating things, configuring things and moving things. Automation should never "clean up". Do your cleanup manually, or this is what you get.

The point of the tooling is that you describe what you want your deploy to look like, and it updates the deploy to match the description. If you delete something from the description and it stays running, that would be very confusing.

I haven’t used their particular tool (ArgoCD) but the ones I have used include an option to keep specific pieces of infrastructure around even when it’s deleted from the description. That’s absolutely what you should be doing for anything that stores data.

Re: Total data loss after botched GitOps and failed backups

#34

Post by gorgoiler » Mon, Aug 28, 2023, 4:21 AM UTC

Kudos to the admin for writing this up and making it public.

Everyone’s lives will involve an irretrievable loss of something, on some scale, at some point. A friend, partner, parent, child — those are the big ones. Your home, job, your pet, a precious object (or in this case, data), or an enjoyment of something: the accumulation can be a lot to handle as time goes on, but it’s also human nature, and it’s important to learn to tackle grief without ignoring it.

Grief is so core to being human that, thankfully, it is an aspect of life where you can find a lot of support. Religious communities, for example, can provide a lot of help that at the same time will be orthogonal to their core mission of worship. You can benefit from the former without having to engage with the latter. Don’t worry about being transactional in looking for help — people will want to help you.

Re: Total data loss after botched GitOps and failed backups

#35

Post by thelittleone » Mon, Aug 28, 2023, 4:21 AM UTC

pnw wrote: post #26

That sucks. Everyone has been there. On my first sysadmin job I did an rm /* as root and then discovered my backups across the network to a remote tape unit had a buffering issue and were basically useless.

Did this at insurance company back in 99. They'd been running on desktop pcs with tops off in a closet (server room) with carpet floors.

Built a cluster in two full racks and Friday after COV started migration. Failed all though Saturday and Sunday to restore data. Has tested backups (DAT drives) after backup which passed but something corrupted. Found another backup tape, and had it up at 0745am on Monday. Zero sleep all weekend had to call in some other admins to get it done. Was both stressful and exhilarating. Became extremely cautious with backup testing since then.

Re: Total data loss after botched GitOps and failed backups

#36

Post by fnord77 » Mon, Aug 28, 2023, 4:22 AM UTC

something like this almost happened where I worked - an atlantis/terraform apply started tearing down our cassandra cluster, which was not backed up.

Caught it in time before it got the volumes

Re: Total data loss after botched GitOps and failed backups

#37

Post by viraptor » Mon, Aug 28, 2023, 4:24 AM UTC

noobermin wrote: post #22

macintux wrote: post #3

The more layers of automation we add, the more invisible points of failure. Magic is great until it isn’t. I feel their pain. The sinking feeling when you realize that data is gone and not coming back is an awful experience.

This is a dumb question, but why can't people just make things simpler? Tools are supposed to make work easier and make things more efficient, but this sort of complexity just seems to hurt, doesn't it?

You can't choose "simpler" overall. The choice is normally between "simpler for one-off things" and "simpler in aggregate". If you choose the first option too often, you'll learn that doing the simplest thing every time paints you into a corner and you have to deal with all the tech debt one day. Doing things in a designed/automated/generated simpler way means you sometimes end up dealing with complex system failures. You can't avoid the complexity - just decide how you organise it.

Re: Total data loss after botched GitOps and failed backups

#38

Post by tedunangst » Mon, Aug 28, 2023, 4:25 AM UTC

pdntspa wrote: post #4

Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.

xkcd: automate, from auto meaning self, and mate meaning...

Re: Total data loss after botched GitOps and failed backups

#39

Post by bee_rider » Mon, Aug 28, 2023, 4:25 AM UTC

noobermin wrote: post #22

macintux wrote: post #3

The more layers of automation we add, the more invisible points of failure. Magic is great until it isn’t. I feel their pain. The sinking feeling when you realize that data is gone and not coming back is an awful experience.

This is a dumb question, but why can't people just make things simpler? Tools are supposed to make work easier and make things more efficient, but this sort of complexity just seems to hurt, doesn't it?

Simpler to use, simpler to learn, simpler to design, and simpler to create are all different properties. Sometimes they are unrelated, sometimes complementary, sometimes in tension.

Re: Total data loss after botched GitOps and failed backups

#40

Post by appplication » Mon, Aug 28, 2023, 4:27 AM UTC

belltaco wrote: post #21

bg24 wrote: post #11

Very sad. But also a miss on admin's part, even if unfortunate. They did not realize during manual restore testing that volume was not being backed up. "Yes and also apparently no. We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim dat…

> manual restore testing They didn't test restores of the actual data being backed up. They did a manual restore test of data that they manually backed up.

I want to feel bad for them, and I do, but really I’m not sure what you expect. They never tested their backup. They tested their ability to manually back up, but I’m not sure why they would ever think to conflate that with automatic processes. If they had automatic backups, why were they relying on manual ones for testing?