Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Total data loss after botched GitOps and failed backups

firefish.social

Post reply on HN
Print view View original

101–110 of 176 posts

Re: Total data loss after botched GitOps and failed backups

#101

Post by tannhaeuser »

sofixa wrote: post #89
TekMol wrote: post #83

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world. None of the following is something which ever crossed my path: - GitOps repository cleanup - yaml manifests that create our namespaces - ArgoCD - Helm deployments - Persistent Volume Claims - Velero - Restic - PVC block volume d…

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite co…

> The elephant in the room is Kubernetes, which [...] often gets used as the go-to even where it doesn't make sense either because it's popular or because that's what people know

Or because folks want to pad their resume with k8s, which is what I'm seeing at work most of the time.

Re: Total data loss after botched GitOps and failed backups

#104

Post by Kwpolska »

sofixa wrote: post #94
Kadin wrote: post #64

Earlier quoted context omitted.

That explains why the K8S defaults are bad, but why use K8S at all here? Personally it seems like a lot of shade-tree server admins are way too eager to bust out much more tooling than is actually necessary for many tasks, just because it's the way that $BIGNAME company does it. Realistically, most people running a personal or small community's IT system aren't going to need the sort of crazy scalability that contain…

You kind of explained it yourself. You need to run a number of components, keep track of all of them, be able to update them, scale them, make sure they're healthy and restarted when needed, etc. This is of course possible in a number of ways, but not trivial. You're basically describing an orchestrator such as Kubernetes or Nomad. Especially with an already existing Helm chart covering all the deployment logic (what…

Mastodon also has manual install instructions: https://docs.joinmastodon.org/admin/install/

They look fairly simple and typical. Monitoring services could be systemd's job, upgrades would involve apt + bundle + yarn.

Re: Total data loss after botched GitOps and failed backups

#105

Post by LoganDark »

msh wrote: post #88
politelemon wrote: post #80

Earlier quoted context omitted.

On similar lines, I like to reformat every year or so at least. It forces a backup test.

Its not a test if failling it results in lost data.

Sure it is. Plenty of companies test load balancing by temporarily shutting off power to one of their data centers and seeing if anyone notices. It's just that usually they can turn the power back on if something does go wrong, but you won't have many recovery options if you blank your only bootable drive. So do use a spare.

Re: Total data loss after botched GitOps and failed backups

#107

Post by nvm0n2 »

sofixa wrote: post #89
TekMol wrote: post #83

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world. None of the following is something which ever crossed my path: - GitOps repository cleanup - yaml manifests that create our namespaces - ArgoCD - Helm deployments - Persistent Volume Claims - Velero - Restic - PVC block volume d…

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite co…

> GitOps is a philosophy which makes sense even on small projects.

Er, does it? Root cause of this catastrophic dataloss incident is that in "GitOps" none of the traditional safety checks can be implemented. In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended. There will also be dry run modes and other helpers.

Because git is intended for source code and not as a way to make stateful changes to servers, there are no features for that. If you push a commit that didn't do what you mean, it will just blindly do it.

It seems like this is a pretty major flaw in the whole "philosophy". The whole point of hacking a VCS into a server admin UI is because people think git will let you roll back infrastructure changes easily. But it cannot, because infrastructure isn't a stateless function of your git repository.

Re: Total data loss after botched GitOps and failed backups

#108

Post by hnarn »

I’m trying very hard to not be flippant here, but I can’t shake the feeling that this Kubernetes norm has to end. I’m not saying Kubernetes needs to disappear, but people need to stop treating it as the new normal, as if VM:s and config management is somehow an outdated and incapable alternative.

To me, this is an example of the complexity of Kubernetes coming back to blow your foot off. Remember, Kubernetes exists to make scaling and redundancy easier, but it’s only easier if you fully understand the implications of every configuration that you make.

Complexity causes incidents, so my mantra will always be that if you propose to introduce complexity, have a justification ready for why its inherent risks are outweighed by the benefits.

If you’re deploying your side projects in an infrastructure this complex, I would strongly suggest taking a step back and questioning if the same benefits really couldn’t be achieved in a simpler way.

Re: Total data loss after botched GitOps and failed backups

#109

Post by jrockway »

speedgoose wrote: post #70
mkl95 wrote: post #66

> We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data This is why for non hobby stuff I advocate for RDS, or its counterpart on your favorite provider. Running a production db on Kubernetes is looking for trouble unless you really,…

I would also advice to have some backups available outside of your favourite cloud provider. Your cloud provider may fuckup, it has happen many times. You may encounter issues such as your account deleted or temporarily unavailable. Or the whole datacenter and many zones are on fire.

For example, the "delete" button on GCP deletes the backups along with the database you're deleting.

I would highly recommend not relying on those backups for anything mission-critical. Transfering the backups to another cloud provider is essential if your risk model involves losing your cloud account somehow.

Re: Total data loss after botched GitOps and failed backups

#110

Post by jrockway »

bomewish wrote: post #98
gabereiser wrote: post #52

Earlier quoted context omitted.

Yeah. Kudos to being open about it but this is an amateur mistake. I ran cloud infrastructure for large companies, small companies, and small companies with large data. Disaster Recovery, Disaster Response, Data Backups, Time to restore, etc are all table stakes to running anything in “production”. They tested their ability to do a backup, but never went through with actually backing it up - automating it - let alone…

but why didn't they just use some normal setup like a postgres database or something? Why all the intense complexity? That makes it less OK

ArgoCD deleting PVCs is pretty much exactly the same as someone clicking around in the web UI deleting EBS volumes (or whatever the equivalent is on their cloud provider). It happens all the time. You have to expect it, or you will lose data.

The only thing worse than automation going haywire is someone in a hurry being careless.

Post reply on HN