Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Total data loss after botched GitOps and failed backups

firefish.social

Post reply on HN
Print view View original

81–90 of 176 posts

Re: Total data loss after botched GitOps and failed backups

#81

Post by EdwardDiego »

ericbarrett wrote: post #71
EdwardDiego wrote: post #56

Earlier quoted context omitted.

It's a very real problem in K8s operators (ArgoCD being a very complicated one, but an operator nonetheless)... ...does absence signify deletion? Some projects decide yes, if the YAML specifying your resource is absent on the next reconciliation, then delete the corresponding K8s resources. Then some projects go with "no, deletion must be explicit, set this flag in your YAML". But the last approach doesn't work as ni…

I think this pattern is fine most of the time. I hate it for PVCs. At the least with cloud-like volumes it should leave a final snapshot that sticks around for a time. I have all the sympathy for the author. I’m sure they don’t feel good right now, but I hope they continue contributing. They’ve learned a hard lesson, but putting it back into their practice is the only way to make it count.

Fully agree.

You can use an admission controller to add guard rails, but I'd hope my GitOps operator also offered those guard rails.

Re: Total data loss after botched GitOps and failed backups

#82

Post by politelemon »

inferiorhuman wrote: post #41
macintux wrote: post #3

The more layers of automation we add, the more invisible points of failure. Magic is great until it isn’t. I feel their pain. The sinking feeling when you realize that data is gone and not coming back is an awful experience.

The more layers of automation we add, the more invisible points of failure. Bullshit. Doing things by hand is more error prone. Without automation this situation would've played out like: well we got used to running things with e.g. --force or --yes or just hitting yes manually at every prompt. Unfortunately we just nuked our data store. Alternatively they would've looked at the dashboards, seen perhaps low CPU or me…

The opposite of complexity is not manual. It is possible to host a setup that is simpler than the one that was chosen.

Re: Total data loss after botched GitOps and failed backups

#83

Post by TekMol »

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world.

None of the following is something which ever crossed my path:

    - GitOps repository cleanup 
    - yaml manifests that create our namespaces
    - ArgoCD
    - Helm deployments
    - Persistent Volume Claims
    - Velero
    - Restic
    - PVC block volume data
    - Vultr
I simply write data to MariaDB (a fork of MySql) and back it up via mysqldump. I wonder if that would have been an option here too?

Re: Total data loss after botched GitOps and failed backups

#84

Post by skoob »

pdntspa wrote: post #4

Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.

By default, ArgoCD doesn't remove resources if the manifests are deleted, but you can switch this by turning on auto prune[1]. Likewise, you can specify whether a persistent volume should be retained or deleted if the k8s PersistentVolume resource is deleted[2]. Both of those settings would have prevented data loss in this case, but of course that's easy to say in hindsight.

[1] https://argo-cd.readthedocs.io/en/stable/user-guide/auto_syn...

[2] https://kubernetes.io/docs/concepts/storage/persistent-volum...

Re: Total data loss after botched GitOps and failed backups

#85

Post by rokkitmensch »

philipswood wrote: post #76
ryanjshaw wrote: post #61

Earlier quoted context omitted.

That was never a backup volume then, it was a mirrored volume. (Ask me how I learned the distinction.)

How did you learn the distinction?

The same way we learn the distinction every night, Pinky. By pissing on the electric fence ourselves!

Re: Total data loss after botched GitOps and failed backups

#86

Post by politelemon »

lamontcg wrote: post #32

This all seems way overly complicated for what could probably be a few services running on a VM or four. Why does Kubernetes/Argo/Helm need to get involved? Why couldn't the whole architecture diagram look a lot more like HN? I feel like we've entirely lost our way with complexity.

It's interesting (so far) that none of the comments have addressed the core question. Why is k8s even needed here?

Many connects are focused on how it could have been solved with the given setup. It's as though using k8s is the default conclusion, and that might actually be the problem. Even in the author's post, their reason for choosing a hosting provider was related to k8s.

I'm not familiar with HN architecture. I agree with your sentiment, IMO there was no need to involve k8s here, keep the implementation simple and grow from there.

Re: Total data loss after botched GitOps and failed backups

#87

Post by gnyman »

SeanAnderson wrote: post #49

Off-topic, but this website has some pretty crazy dark-patterns! I went to the user's profile and right-clicked their photo. The contextmenu action is intercepted and replaced with a custom menu mimicking the browser's context menu. This fake context menu has the option "Open in Window". I click it. The website doubles-down on the charade and opens a "popup" inside itself - maximize/minimize icons in the action bar a…

That's not a "dark pattern", it's just the result of of making the browser a OS and giving websites free rein of anything inside.

There is an aptly named browser extension to help combat this called StopTheMadness [1] , I recommend it.

Let's try to save the term Dark Patterns for the things that that deserve it, where the goal is to steer the user to make a unfavourable choice at the benefit of someone else.

https://underpassapp.com/StopTheMadness/

Re: Total data loss after botched GitOps and failed backups

#88

Post by msh »

politelemon wrote: post #80
j7ake wrote: post #51

Earlier quoted context omitted.

Buy an extra computer that is blank and periodically restore your back ups into the extra computer.

On similar lines, I like to reformat every year or so at least. It forces a backup test.

Its not a test if failling it results in lost data.

Re: Total data loss after botched GitOps and failed backups

#89

Post by sofixa »

TekMol wrote: post #83

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world. None of the following is something which ever crossed my path: - GitOps repository cleanup - yaml manifests that create our namespaces - ArgoCD - Helm deployments - Persistent Volume Claims - Velero - Restic - PVC block volume d…

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree.

Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects.

None of those are "wrong" or "overcomplicated" options.

The elephant in the room is Kubernetes, which is indeed quite complex, and often gets used as the go-to even where it doesn't make sense [1] either because it's popular or because that's what people know, or because of the ready-made tools from others (e.g. if all you want to deploy exists in the form of Helm charts, it can save you lots of time) but it has its place and brings a lot to the table. You just have to be aware of the risks the complexity brings.

Disclaimer: I work at HashiCorp, I'm a massive fan of Nomad and think it's a better fit than Kubernetes in many cases, but dismissing Kubernetes outright is wrong.

1 - https://atodorov.me/img/nomad/kubernetes.jpg#center

Re: Total data loss after botched GitOps and failed backups

#90

Post by trustingtrust »

pech0rin wrote: post #75

Very sorry to hear about this. Working in ops for 10+ years I know that drop kick in the stomach feeling when everything is going wrong and nothing is working as expected. I hope the author continues forward and takes this as a lesson. My own two cents which others have echoed is this is extremely overly complex setup for this situation. Its been told 100x times on HN about this but there is a reason. The more comple…

> Hope as an industry someday we can get back to simplicity

The services upon services with no idea what underlying layers are doing and pretty much enterprises collaborating to get stuff 'out the door' means we may not go back to simplicity easily. At this point I am only finding work who are basically middleware and rarely doing something unique even there. Basically getting contracts and servicing a business need is all that I see when I try to find jobs these days. Initially I was doing something quite unique in networking but now that the client needs have changed its basically just another product that is servicing clients which big companies like Arista can't do for them as there are custom requirements.

Post reply on HN