Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Total data loss after botched GitOps and failed backups

firefish.social

Post reply on HN
Print view View original

71–80 of 176 posts

Re: Total data loss after botched GitOps and failed backups

#71

Post by ericbarrett »

EdwardDiego wrote: post #56
pdntspa wrote: post #4

Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.

It's a very real problem in K8s operators (ArgoCD being a very complicated one, but an operator nonetheless)... ...does absence signify deletion? Some projects decide yes, if the YAML specifying your resource is absent on the next reconciliation, then delete the corresponding K8s resources. Then some projects go with "no, deletion must be explicit, set this flag in your YAML". But the last approach doesn't work as ni…

I think this pattern is fine most of the time. I hate it for PVCs. At the least with cloud-like volumes it should leave a final snapshot that sticks around for a time.

I have all the sympathy for the author. I’m sure they don’t feel good right now, but I hope they continue contributing. They’ve learned a hard lesson, but putting it back into their practice is the only way to make it count.

Re: Total data loss after botched GitOps and failed backups

#72

Post by accrual »

devmor wrote: post #9

Automation is wonderful for creating things, configuring things and moving things. Automation should never "clean up". Do your cleanup manually, or this is what you get.

Although the scope is smaller and therefore should be safer, it is still necessary to purge old logs and old backups, etc. in some places.

When I need to do this, I make sure there's a specific /logs or /backups folder where only certain filetypes of a certain modified dates are deleted.

Re: Total data loss after botched GitOps and failed backups

#74

Post by devmor »

sp332 wrote: post #18
devmor wrote: post #9

Automation is wonderful for creating things, configuring things and moving things. Automation should never "clean up". Do your cleanup manually, or this is what you get.

The point of the tooling is that you describe what you want your deploy to look like, and it updates the deploy to match the description. If you delete something from the description and it stays running, that would be very confusing.

What you're describing as "confusing" is exactly how every CI tooling I've worked with functions.

You describe what you want your deploy to look like, and that's what gets deployed. It doesn't start removing/deleting things to match.

It might take resources offline or leave them orphaned, but never delete them.

Re: Total data loss after botched GitOps and failed backups

#75

Post by pech0rin »

Very sorry to hear about this. Working in ops for 10+ years I know that drop kick in the stomach feeling when everything is going wrong and nothing is working as expected. I hope the author continues forward and takes this as a lesson.

My own two cents which others have echoed is this is extremely overly complex setup for this situation. Its been told 100x times on HN about this but there is a reason. The more complex a system is the more points of failure you have. For all the tools you were running you should have had a team of ops people. In reality a database, some servers and a load balancer would get you 99% of the way there.

Modern engineering is a dumpster fire of complexity mostly hocked by shills working to sell contracts to enterprises.

You have learned this lesson the hard way but less moving parts means less things that can go wrong. Basics like monitoring, backup, testing etc go only so far if your system is a rube goldberg machine. Hope as an industry someday we can get back to simplicity

Re: Total data loss after botched GitOps and failed backups

#76

Post by philipswood »

ryanjshaw wrote: post #61
defaultcompany wrote: post #25

Earlier quoted context omitted.

Ugh I did something like this once when I wrote a script to rsync one volume to a backup volume. Everything was great until someone accidentally rm -r ‘d the source volume and the script happily deleted everything on the backup volume to keep it in sync.

That was never a backup volume then, it was a mirrored volume. (Ask me how I learned the distinction.)

How did you learn the distinction?

Re: Total data loss after botched GitOps and failed backups

#78

Post by tetha »

livueta wrote: post #28
noobermin wrote: post #22

Earlier quoted context omitted.

This is a dumb question, but why can't people just make things simpler? Tools are supposed to make work easier and make things more efficient, but this sort of complexity just seems to hurt, doesn't it?

I wonder if it has to do with mismatches between -as-cattle and -as-pets philosophies and usecases. From the post, it looks like they were using k8s. It's hard to tell given the, you know, total data loss, but it looks like the instance had at max triple-digit users. In my experience running small community stuff like this, the -as-cattle tooling, even though I'm familiar with it from work, is way overkill and introd…

At work, we're running a hybrid container/vm setup. We host the applications from our dev-teams as containers, since there is a lot of them and this allows them to do most of the work to integrate new apps with the stack. However, the actual data stores like postgres, glusterfs and such are simple VMs for a simple reason: It has less failure modes and many of the failure modes on a VM are less subtle and strange to deal with. And very few failure modes on a VM go straight to irrecoverable dataloss.

In fact, the company I work at has deployed production like that for many years when our needs were simpler with just 1-2 applications. 1-4 application VMs provisioned by a config management, maybe a loadbalancer, maybe 1-2 DB VMs behind it. Worked like a charm until the new parent company started throwing loads of complexity at it.

Re: Total data loss after botched GitOps and failed backups

#79

Post by marcus_holmes »

mkl95 wrote: post #66

> We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data This is why for non hobby stuff I advocate for RDS, or its counterpart on your favorite provider. Running a production db on Kubernetes is looking for trouble unless you really,…

We had a database downtime on RDS. I queried it, because we were paying good money for a failover database. We were told that the downtime was because the database was being updated, and all databases are upgraded at the same time so the failover was also being upgraded.

Never using RDS again. I'll stick with running Postgres on something I can actually manage myself.

Re: Total data loss after botched GitOps and failed backups

#80

Post by politelemon »

j7ake wrote: post #51
nicbou wrote: post #27

Earlier quoted context omitted.

I know that this is the time to say "you should have tested your backups", but I never tested mine. How does one test a (personal) backup strategy?

Buy an extra computer that is blank and periodically restore your back ups into the extra computer.

On similar lines, I like to reformat every year or so at least. It forces a backup test.
Post reply on HN