Total data loss after botched GitOps and failed backups • Hacker News Board

Re: Total data loss after botched GitOps and failed backups

#161

Post by Gud » Mon, Aug 28, 2023, 7:00 PM UTC

diarrhea wrote: post #133

Gud wrote: post #128

Earlier quoted context omitted.

Reduce your abstractions and reduce your dependencies for a more stable environment. It will give you greater control while you at the same time don’t need to relearn everything every 5 years. Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine? This fatal scenario could have been easily avoided with a basic rsync cronjob to rsync.net or another $10 vi…

> Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine? I am very interested in this upcoming space. I had set up devbox.io for local development, and it even offers the concept of services. Like a docker compose for local development, just directly on the host. Nasty and hairy stuff like Python is still best shoved into a container to be done with it,…

I believe this gigantic ecosystem of build tools is there because the solution itself is needlessly complex.

I also believe using python for web development is a mistake.

Re: Total data loss after botched GitOps and failed backups

#162

Post by mdaniel » Mon, Aug 28, 2023, 7:08 PM UTC

fnord77 wrote: post #36

something like this almost happened where I worked - an atlantis/terraform apply started tearing down our cassandra cluster, which was not backed up. Caught it in time before it got the volumes

You're in good company: https://news.ycombinator.com/item?id=36665004

The ol' adage "to err is human, to wipe out production requires automation terraform"

Re: Total data loss after botched GitOps and failed backups

#163

Post by nighmi » Mon, Aug 28, 2023, 7:19 PM UTC

kuchenbecker wrote: post #17

pdntspa wrote: post #4

Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.

Yeah, it's a hard lesson. We would always safe-expire or soft-delete to avoid this so each operation is undo-able for ~2 weeks.

How do you set this up?

Re: Total data loss after botched GitOps and failed backups

#164

Post by kuchenbecker » Mon, Aug 28, 2023, 7:31 PM UTC

nighmi wrote: post #163

kuchenbecker wrote: post #17

Earlier quoted context omitted.

Yeah, it's a hard lesson. We would always safe-expire or soft-delete to avoid this so each operation is undo-able for ~2 weeks.

How do you set this up?

In a nutshell, we "abandon" the resource and clean up abandoned resources after N days. To delete something, you need to explicitly check it's name into a list of resources to delete.

So deletions are explicit and a mistake is just un-abandoning.

Re: Total data loss after botched GitOps and failed backups

#165

Post by depereo » Mon, Aug 28, 2023, 7:51 PM UTC

VectorLock wrote: post #54

depereo wrote: post #47

Earlier quoted context omitted.

Velero can be configured to run on a schedule, but the scheduled command was apparently not the exact same command they were using to perform the manual tests - the scheduled job was missing part of the command, basically.

Sp they were manually doing a backup and then testing that backup, rather than testing their automated backups? If thats what they were doing that just makes me wonder... why?

It's possible that the backup restore/test process was still 'automated' to some degree, like a manually-run pipeline or playbook etc.

Obviously there were mistakes made in process design and tools deployment but I don't think this is 'baffling incompetence' more than it is a couple of small mistakes that compounded each other to have a pretty spectacular impact.

Re: Total data loss after botched GitOps and failed backups

#166

Post by earthling8118 » Mon, Aug 28, 2023, 8:19 PM UTC

sofixa wrote: post #89

TekMol wrote: post #83

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world. None of the following is something which ever crossed my path: - GitOps repository cleanup - yaml manifests that create our namespaces - ArgoCD - Helm deployments - Persistent Volume Claims - Velero - Restic - PVC block volume d…

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite co…

Nomad is honestly really good. For a long time I've wished that it had a wider reach because I just outright don't want to go back to Kubernetes after using them both.

I'm really fearful that the recent events have harmed the chances of that happening though. It's a shame so I hope that isn't how it all happens.

Re: Total data loss after botched GitOps and failed backups

#167

Post by sangnoir » Mon, Aug 28, 2023, 8:32 PM UTC

pech0rin wrote: post #75

Very sorry to hear about this. Working in ops for 10+ years I know that drop kick in the stomach feeling when everything is going wrong and nothing is working as expected. I hope the author continues forward and takes this as a lesson. My own two cents which others have echoed is this is extremely overly complex setup for this situation. Its been told 100x times on HN about this but there is a reason. The more comple…

> Modern engineering is a dumpster fire of complexity mostly hocked by shills working to sell contracts to enterprises.

I think that's like 10% of the reason. 80% is following hype(-), resume-driven engineering (-), and or standardizing on non-proprietary tools to improve job mobility(+). Sure, maybe maintaining the esoteric DSL created by my employer's custom load balancer and DB failover codebases named after obscure comic book characters[1] may arguably be the "right" thing to do - or I could advocate for k8s which may be a little overkill for our needs, but is easier to onboard new joiners, and once can actually search for solutions on StackOverflow. Kubernetes is also a jack of all trades and is designed to handle configurations much more complex than I have, but at least it's "portable".

1. "Oh, _La'varo_ is named after a villain that appeared in a 1959 issue of Superman (Volume 7), it once was an LB that avoided hot instances when routing requests, but now its KV store"

Re: Total data loss after botched GitOps and failed backups

#168

Post by rdg42 » Mon, Aug 28, 2023, 9:22 PM UTC

smartbit wrote: post #69

havnagiggle wrote: post #46

Earlier quoted context omitted.

IMO it's because of this default in k8s storage class: https://kubernetes.io/docs/concepts/storage/storage-classes/... . Almost everything in a k8s cluster can be ephemeral... except your data! But they make the defaults insane and it eats a lot of people's lunches. It's honestly very sad. I fortunately haven't "lost" data in 25 years, but I still remember the pain when I made this kind of mistake.

The default PVC reclaim policy is Delete https://kubernetes.io/docs/reference/kubernetes-api/config-a... . The PV will be retained after deletion of the PVC by adding persistentVolumeReclaimPolicy: Retain Alternatively set the reclaimPolicy on the StorageClass https://kubernetes.io/docs/concepts/storage/storage-classes/... for retaining all PVs of that SC after deletion of a PVC: reclaimPolicy: Retain If Lily Cohen w…

Thank you! Had to scroll way too much to read that.

On production systems "reclaimPolicy: Retain" on the storageClass feels like a no brainer to (mostly) avoid such disaster.

Re: Total data loss after botched GitOps and failed backups

#169

Post by Anon_Forever » Mon, Aug 28, 2023, 9:33 PM UTC

fnord77 wrote: post #36

something like this almost happened where I worked - an atlantis/terraform apply started tearing down our cassandra cluster, which was not backed up. Caught it in time before it got the volumes

Terraform has had the "prevent_destroy" functionality for quite some time. Highly suggest using it for everything you're going to miss if it's gone.

[0] https://developer.hashicorp.com/terraform/language/meta-argu...

Re: Total data loss after botched GitOps and failed backups

#170

Post by zeeZ » Mon, Aug 28, 2023, 10:15 PM UTC

ericbarrett wrote: post #71

EdwardDiego wrote: post #56

Earlier quoted context omitted.

It's a very real problem in K8s operators (ArgoCD being a very complicated one, but an operator nonetheless)... ...does absence signify deletion? Some projects decide yes, if the YAML specifying your resource is absent on the next reconciliation, then delete the corresponding K8s resources. Then some projects go with "no, deletion must be explicit, set this flag in your YAML". But the last approach doesn't work as ni…

I think this pattern is fine most of the time. I hate it for PVCs. At the least with cloud-like volumes it should leave a final snapshot that sticks around for a time. I have all the sympathy for the author. I’m sure they don’t feel good right now, but I hope they continue contributing. They’ve learned a hard lesson, but putting it back into their practice is the only way to make it count.

You can set a PVC's reclaim policy to "Retain" (default is "Delete"). When the namespaced claim is deleted, the volume will still be around until it is manually deleted. The docs even suggest to do this for "precious data".