Total data loss after botched GitOps and failed backups • Hacker News Board

Re: Total data loss after botched GitOps and failed backups

#151

Post by omniglottal » Mon, Aug 28, 2023, 4:39 PM UTC

EdwardDiego wrote: post #81

ericbarrett wrote: post #71

Earlier quoted context omitted.

I think this pattern is fine most of the time. I hate it for PVCs. At the least with cloud-like volumes it should leave a final snapshot that sticks around for a time. I have all the sympathy for the author. I’m sure they don’t feel good right now, but I hope they continue contributing. They’ve learned a hard lesson, but putting it back into their practice is the only way to make it count.

Fully agree. You can use an admission controller to add guard rails, but I'd hope my GitOps operator also offered those guard rails.

It does. The option is called "prune" and absent resources will be deleted ("pruned") when this is set to "true".

Re: Total data loss after botched GitOps and failed backups

#152

Post by hdjjhhvvhga » Mon, Aug 28, 2023, 4:40 PM UTC

nvm0n2 wrote: post #107

sofixa wrote: post #89

Earlier quoted context omitted.

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite co…

> GitOps is a philosophy which makes sense even on small projects. Er, does it? Root cause of this catastrophic dataloss incident is that in "GitOps" none of the traditional safety checks can be implemented. In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended. There…

> If the application is containerised, like most are these days, how do you propose running these on said VM?

It really depends on how you implement it. In simplest setups - yes, you can destroy infrastructure with data and when you recreate the infra the data is gone. But the implementations I worked on were specifically designed to withstand this kind of problem.

Re: Total data loss after botched GitOps and failed backups

#153

Post by WorldMaker » Mon, Aug 28, 2023, 5:07 PM UTC

nvm0n2 wrote: post #107

sofixa wrote: post #89

Earlier quoted context omitted.

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite co…

> GitOps is a philosophy which makes sense even on small projects. Er, does it? Root cause of this catastrophic dataloss incident is that in "GitOps" none of the traditional safety checks can be implemented. In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended. There…

GitOps workflows can have safety gates that require a manual double-check/approval step, for instance if the deployment plan is substantially different or if deletes and loss of data would occur. They weren't implemented in this case, but that doesn't mean they can't be implemented. They probably should be implemented as a takeaway from this article.

Source control is great at auditing why a change occurred. (Keep in mind some of those changes include things like bad merge accidents and sloppy refactors. Source control never guarantees a perfect state of the code at any point in time, only a saved state.) Source control can also give you an estimate in how big of a change occurred (diff size, number of files changed/moved/deleted). You can use those same tools in the process of a GitOps workflow and in setting smart manual gates, not just in post-mortem root cause finding when things go wrong.

Re: Total data loss after botched GitOps and failed backups

#154

Post by lazylion2 » Mon, Aug 28, 2023, 5:25 PM UTC

SeanAnderson wrote: post #49

Off-topic, but this website has some pretty crazy dark-patterns! I went to the user's profile and right-clicked their photo. The contextmenu action is intercepted and replaced with a custom menu mimicking the browser's context menu. This fake context menu has the option "Open in Window". I click it. The website doubles-down on the charade and opens a "popup" inside itself - maximize/minimize icons in the action bar a…

On Firefox shift+right-click forces regular contextmenu

Re: Total data loss after botched GitOps and failed backups

#155

Post by figassis » Mon, Aug 28, 2023, 5:55 PM UTC

pech0rin wrote: post #75

Very sorry to hear about this. Working in ops for 10+ years I know that drop kick in the stomach feeling when everything is going wrong and nothing is working as expected. I hope the author continues forward and takes this as a lesson. My own two cents which others have echoed is this is extremely overly complex setup for this situation. Its been told 100x times on HN about this but there is a reason. The more comple…

My philosophy here is never automate/gitops something that you cannot revert manually if something goes wrong. I learned this almost the same way as OP, but in test and dev environments. As my K8s clusters crashed irrecoverably in many different ways (etcd, pvcs, velero, hdd failures, shared storage, databases) to the point where they'd need to be rebuilt, I imagined what it would be like in prod, and the thought made me quickly decide to:

- Not host databases inside k8s - Not host shared storage or storage clusters in k8s - Drop etcd, use external postgres as control plane database - Use velero, but make sure you do no need to rely on velero (backup manifests, not databases or volumes) - Also keep your manifests in git - Generaly treat your entire cluster as ephemeral and optimize for full rebuilds (drive your time to rebuild down to a few hours or minutes).

Re: Total data loss after botched GitOps and failed backups

#156

Post by lamontcg » Mon, Aug 28, 2023, 5:56 PM UTC

politelemon wrote: post #86

lamontcg wrote: post #32

This all seems way overly complicated for what could probably be a few services running on a VM or four. Why does Kubernetes/Argo/Helm need to get involved? Why couldn't the whole architecture diagram look a lot more like HN? I feel like we've entirely lost our way with complexity.

It's interesting (so far) that none of the comments have addressed the core question. Why is k8s even needed here? Many connects are focused on how it could have been solved with the given setup. It's as though using k8s is the default conclusion, and that might actually be the problem. Even in the author's post, their reason for choosing a hosting provider was related to k8s. I'm not familiar with HN architecture. I…

All the followups pretty much just talked right over the point I was making.

Re: Total data loss after botched GitOps and failed backups

#157

Post by hotsauceror » Mon, Aug 28, 2023, 6:03 PM UTC

pdntspa wrote: post #4

Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.

Agreed. Reading this kind of thing makes my blood run cold. Your 'accidental sudo rm -rf /' not only destroyed one machine, but all of them. The entire cluster, standbys, backups, storage volumes, the whole enchilada. That's a Business Ending Event, in one merge request.

I'm hard-pressed to think of a scenario where a single sysadmin in an 'old-fashioned' enterprise datacenter could do as much damage with, maybe not exactly a one-liner but a lightweight change instruction.

The risk-reward calculation seems completely bananas to me.

Re: Total data loss after botched GitOps and failed backups

#158

Post by hnarn » Mon, Aug 28, 2023, 6:10 PM UTC

hdjjhhvvhga wrote: post #150

hnarn wrote: post #123

Earlier quoted context omitted.

It's a very complex discussion, almost complex enough to be unsuitable for text, because for every reply ten more "what-ifs" seem to open up. It's hard to be general enough that what you're saying is useful, while being specific enough that you're actually answering the question. If we're talking about a multi-container application with moving parts, then yes, Kubernetes is suitable exactly because it was created to…

You have omitted the crux of the issue: the whole point of GitOps is that you can always roll your deployments back, to any particular commit if needed. The very fact that GitOps was used against itself means it was set up incorrectly.

I never even mentioned gitops, and furthermore (to my point), gitops principles do not require kubernetes.

Re: Total data loss after botched GitOps and failed backups

#159

Post by zzyzxd » Mon, Aug 28, 2023, 6:26 PM UTC

Here we go again, a lot of comments saying k8s and gitops are too complex.

IMHO, your ops team can accidentally delete your data one way or another regardless. Today it was moving files to a wrong directory in git, yesterday it could have been human executing `rm -rf` against the wrong path. The highlighted item here should be the bad backup process. You can't say your data is backed up unless you actually verify successful restoration from the backup data, on a regular basis. This was true 20 years ago for your SQL database instance, and this is still true today for your k8s PVCs.

FWIW, I have been running GitOps long enough that any PR that moves files around raise the highest alerts in my head. The fundamental issue I have seen in many places is that engineers stores infra code in git and call it "GitOps". When you adopt GitOps concept, the most important thing is to train your engineers to switch to a different mental model, where your git repo is the _desired_ state of your infra -- it's not the actual state of the infra, and it's not a store of your imperative commands to manipulate the infra. When your desired state of your namespace is to not to exist, your gitops engine will try to make that happen!

Re: Total data loss after botched GitOps and failed backups

#160

Post by ianlevesque » Mon, Aug 28, 2023, 6:46 PM UTC

JeremyNT wrote: post #135

pech0rin wrote: post #75

Very sorry to hear about this. Working in ops for 10+ years I know that drop kick in the stomach feeling when everything is going wrong and nothing is working as expected. I hope the author continues forward and takes this as a lesson. My own two cents which others have echoed is this is extremely overly complex setup for this situation. Its been told 100x times on HN about this but there is a reason. The more comple…

You're right of course that complexity makes it harder to reason about, which makes it easy to "fat finger" a destructive operation with unintended consequences. However I'd say the truly disastrous issue here wasn't really that. Data loss is something we've always had to plan around, whether due to mechanical failure, bug, or operator error. The actual death knell here was a classic ops blunder: > From what I had se…

Yeah there are always sharp edges on backup processes. For example I'd been taking faithful backups of Hyper-V virtual machines, but apparently there's a private key that is stored elsewhere (and thus wasn't backed up), but without which you can't restore the software TPMs (~required since Windows 11). There are so many of these that the only way to really know is to test. Thankfully that is becoming so much easier with all the declarative infrastructure, Docker/containerization, and virtualization tools these days. Spin up a second copy of your infrastructure once in a while and try a restore.