Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Total data loss after botched GitOps and failed backups

firefish.social

Post reply on HN
Print view View original

131–140 of 176 posts

Re: Total data loss after botched GitOps and failed backups

#131

Post by gabereiser »

kikimora wrote: post #125
jrockway wrote: post #110

Earlier quoted context omitted.

ArgoCD deleting PVCs is pretty much exactly the same as someone clicking around in the web UI deleting EBS volumes (or whatever the equivalent is on their cloud provider). It happens all the time. You have to expect it, or you will lose data. The only thing worse than automation going haywire is someone in a hurry being careless.

Most of production EBS volumes will be with deletion protection on. Also AWS asks you to explicitly type “delete” when you delete things. I’d blame cloud provider for making it too easy to destroy data and not having built-in backups. But then this is what cheap cloud providers do.

Vultr has volume protection too but only when in-use. During a deployment you can easily do what the OP did if you don’t do Canary deployments with red/blue failover for zero downtime. The volume will become “non-in-use” and then…

The real issue here is it all came down to a tab.

“How the backup was created was the key component. Manual backups were made with a command and recurring were setup with a yaml manifest. When we ran manual backups for testing, we set the default-volumes-to-fs-backup flag. The yaml manifest HAD the flag, but it was indented one two many times, so it wasn't actually setting the correct key in the velero helm chart, which might as well have not been setting it at all.”

Re: Total data loss after botched GitOps and failed backups

#133

Post by diarrhea »

Gud wrote: post #128
CommanderData wrote: post #117

Earlier quoted context omitted.

If the application is containerised, like most are these days, how do you propose running these on said VM? If we're talking about something a little more complicated then a single container, then you arrive at the same problem Kubernetes is solving.

Reduce your abstractions and reduce your dependencies for a more stable environment. It will give you greater control while you at the same time don’t need to relearn everything every 5 years. Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine? This fatal scenario could have been easily avoided with a basic rsync cronjob to rsync.net or another $10 vi…

> Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine?

I am very interested in this upcoming space. I had set up devbox.io for local development, and it even offers the concept of services. Like a docker compose for local development, just directly on the host.

Nasty and hairy stuff like Python is still best shoved into a container to be done with it, but even there we are seeing a move to venv by default (Debian 12+), which should solve a lot of issues, making *nix more viable again.

So I guess the main distinction so far is that containers are more performant, at less isolation than VMs, and containers have a gigantic ecosystem of build tools. It's easy to build a OCI image. Everyone and their mother has put together a Dockerfile by now. Not so with VMs. That might change if nix catches on and we get proper tools to fully describe VMs a priori. I reckon technology is already good enough to make VM startups performant enough (Firecracker, ...)?

Re: Total data loss after botched GitOps and failed backups

#134

Post by diarrhea »

belltaco wrote: post #21
bg24 wrote: post #11

Very sad. But also a miss on admin's part, even if unfortunate. They did not realize during manual restore testing that volume was not being backed up. "Yes and also apparently no. We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim dat…

> manual restore testing They didn't test restores of the actual data being backed up. They did a manual restore test of data that they manually backed up.

Ah. That part was unclear from the text. Thanks for clarifying.

Re: Total data loss after botched GitOps and failed backups

#135

Post by JeremyNT »

pech0rin wrote: post #75

Very sorry to hear about this. Working in ops for 10+ years I know that drop kick in the stomach feeling when everything is going wrong and nothing is working as expected. I hope the author continues forward and takes this as a lesson. My own two cents which others have echoed is this is extremely overly complex setup for this situation. Its been told 100x times on HN about this but there is a reason. The more comple…

You're right of course that complexity makes it harder to reason about, which makes it easy to "fat finger" a destructive operation with unintended consequences.

However I'd say the truly disastrous issue here wasn't really that. Data loss is something we've always had to plan around, whether due to mechanical failure, bug, or operator error. The actual death knell here was a classic ops blunder:

> From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data. While manual backup and restore tests were run once a month to ensure our backups were functioning, they were run manually. After digging into why our restores were not coming up with data, I found that our recurring backups were missing the flag to run volume backups with Restic which snapshots PVC block volume data.

So they weren't actually backing up the data they thought they were, which is a class of error that is familiar to everybody who has been working in ops for a while. Unless you test the restore of an actual backup process, you have no idea whether you have useful backups.

Re: Total data loss after botched GitOps and failed backups

#136

Post by evanelias »

ryanjshaw wrote: post #65
EdwardDiego wrote: post #56

Earlier quoted context omitted.

It's a very real problem in K8s operators (ArgoCD being a very complicated one, but an operator nonetheless)... ...does absence signify deletion? Some projects decide yes, if the YAML specifying your resource is absent on the next reconciliation, then delete the corresponding K8s resources. Then some projects go with "no, deletion must be explicit, set this flag in your YAML". But the last approach doesn't work as ni…

Declarative SQL version control has the same challenge. Best practice now seems to be moving towards explicit SQL migration scripts.

Guardrails are essential. A properly-designed declarative schema change tool provides guardrails to detect and prevent unwanted deletion / drops / lossy conversions. I develop widely-used software in this space (Skeema) and my tool has always offered drop-prevention functionality ever since its first beta release in 2016.

I also argue that guardrails are equally important for imperative migration tools as well, but more often they're lacking or half-baked, which gives a false sense of safety. For example, down/reverse migrations are a very common landmine for human error. Order-of-operations problems also happen frequently when imperative tools are used by large dev teams, resulting in subtle schema drift between environments when there's disagreement between lexicographic migration file order, git history order, and the actual migration application order on each DB.

Re: Total data loss after botched GitOps and failed backups

#137

Post by evanelias »

marcus_holmes wrote: post #79
mkl95 wrote: post #66

> We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data This is why for non hobby stuff I advocate for RDS, or its counterpart on your favorite provider. Running a production db on Kubernetes is looking for trouble unless you really,…

We had a database downtime on RDS. I queried it, because we were paying good money for a failover database. We were told that the downtime was because the database was being updated, and all databases are upgraded at the same time so the failover was also being upgraded. Never using RDS again. I'll stick with running Postgres on something I can actually manage myself.

Was it a major version upgrade? Postgres WAL replication is a physical replication stream which doesn't work across major versions. I don't know much about RDS Postgres but I'd assume this is the reason. Logical replication can solve this, but has some important limitations (e.g. no replication of DDL), so it's understandable that RDS can't leverage this generically and automatically.

I would expect that RDS MySQL doesn't have this problem since MySQL's built-in replication is logical rather than physical.

Re: Total data loss after botched GitOps and failed backups

#139

Post by isitmadeofglass »

CommanderData wrote: post #117
hnarn wrote: post #108

I’m trying very hard to not be flippant here, but I can’t shake the feeling that this Kubernetes norm has to end. I’m not saying Kubernetes needs to disappear, but people need to stop treating it as the new normal, as if VM:s and config management is somehow an outdated and incapable alternative. To me, this is an example of the complexity of Kubernetes coming back to blow your foot off. Remember, Kubernetes exists t…

If the application is containerised, like most are these days, how do you propose running these on said VM? If we're talking about something a little more complicated then a single container, then you arrive at the same problem Kubernetes is solving.

[dead]

Re: Total data loss after botched GitOps and failed backups

#140

Post by mannyv »

tetha wrote: post #122
mannyv wrote: post #100

Earlier quoted context omitted.

Terraform will do this. It's just like a command line, but for your environment.

Terraform, unless invoked as tf-yolo (aka terraform -auto-approve), will present you with a cheerful "Hello, I would like to delete everything. Continue: y/n". This makes it fairly easy to panic before deleting everything. Otoh, if you run and don't read the plan carefully... eh. We're planning to streamline our run approvals a bit, because approving every DNS record addition results in some approval fatigue, but res…

True. I carefully look over tf apply to ensure it's doing what i want. But other people are problematic.
Post reply on HN