Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.
Ugh I did something like this once when I wrote a script to rsync one volume to a backup volume. Everything was great until someone accidentally rm -r ‘d the source volume and the script happily deleted everything on the backup volume to keep it in sync.
Total data loss after botched GitOps and failed backups
61–70 of 176 posts
Re: Total data loss after botched GitOps and failed backups
#62The only data that I really care about are my photos and videos and they are backed up to four different providers. But this also made me realize how upset I would be if my blog over at micro.blog was accidentally deleted. It’s more of a journal of my digital nomadding with my wife across the US than anything else. I immediately went over and started a JSON export in bar (?) format.
Re: Total data loss after botched GitOps and failed backups
#63Automation is wonderful for creating things, configuring things and moving things. Automation should never "clean up". Do your cleanup manually, or this is what you get.
The point of the tooling is that you describe what you want your deploy to look like, and it updates the deploy to match the description. If you delete something from the description and it stays running, that would be very confusing.
Re: Total data loss after botched GitOps and failed backups
#64This all seems way overly complicated for what could probably be a few services running on a VM or four. Why does Kubernetes/Argo/Helm need to get involved? Why couldn't the whole architecture diagram look a lot more like HN? I feel like we've entirely lost our way with complexity.
IMO it's because of this default in k8s storage class: https://kubernetes.io/docs/concepts/storage/storage-classes/... . Almost everything in a k8s cluster can be ephemeral... except your data! But they make the defaults insane and it eats a lot of people's lunches. It's honestly very sad. I fortunately haven't "lost" data in 25 years, but I still remember the pain when I made this kind of mistake.
Personally it seems like a lot of shade-tree server admins are way too eager to bust out much more tooling than is actually necessary for many tasks, just because it's the way that $BIGNAME company does it.
Realistically, most people running a personal or small community's IT system aren't going to need the sort of crazy scalability that container orchestration systems are designed to provide. And if you're not running them at significant scale, the containerization and orchestration-system overhead are often quite significant fractions of the overall resource footprint. Plus there's the unnecessary complexity due to all the levels of abstraction that just don't need to be there.
E.g. for a Mastodon instance, I wouldn't touch Kubernetes or complex orchestration unless I was out of other, more traditional options for scaling. The server side is already broken into a number of components (RoR app, PosgreSQL, Redis, Sidekiq, node.js) which you can separate out onto their own servers, and from there each component has preferred ways of scaling based on need. And while nothing is safe from failure, backing up a bog-standard PostgreSQL VPS is a lot more straightforward than K8S.
If you're doing deployments dozens of times a day, of course automation is desirable. But if you're doing it once, I suspect most people would be hard-pressed to make back the investment of time and additional testing required (well, should be required) of working through a complex container-management architecture.
Re: Total data loss after botched GitOps and failed backups
#65Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.
It's a very real problem in K8s operators (ArgoCD being a very complicated one, but an operator nonetheless)... ...does absence signify deletion? Some projects decide yes, if the YAML specifying your resource is absent on the next reconciliation, then delete the corresponding K8s resources. Then some projects go with "no, deletion must be explicit, set this flag in your YAML". But the last approach doesn't work as ni…
Re: Total data loss after botched GitOps and failed backups
#66This is why for non hobby stuff I advocate for RDS, or its counterpart on your favorite provider. Running a production db on Kubernetes is looking for trouble unless you really, really know what you are doing.
Re: Total data loss after botched GitOps and failed backups
#67Off-topic, but this website has some pretty crazy dark-patterns! I went to the user's profile and right-clicked their photo. The contextmenu action is intercepted and replaced with a custom menu mimicking the browser's context menu. This fake context menu has the option "Open in Window". I click it. The website doubles-down on the charade and opens a "popup" inside itself - maximize/minimize icons in the action bar a…
Might be related: I recently joined a company using Flutter web and notices similar stuff. Yes, the software engineers are wondering: "Why do we reimplement browser stuff?" But the business ask for it and we don't hear complaints from the end user.
Re: Total data loss after botched GitOps and failed backups
#68Earlier quoted context omitted.
Velero can be configured to run on a schedule, but the scheduled command was apparently not the exact same command they were using to perform the manual tests - the scheduled job was missing part of the command, basically.
Sp they were manually doing a backup and then testing that backup, rather than testing their automated backups? If thats what they were doing that just makes me wonder... why?
Re: Total data loss after botched GitOps and failed backups
#69This all seems way overly complicated for what could probably be a few services running on a VM or four. Why does Kubernetes/Argo/Helm need to get involved? Why couldn't the whole architecture diagram look a lot more like HN? I feel like we've entirely lost our way with complexity.
IMO it's because of this default in k8s storage class: https://kubernetes.io/docs/concepts/storage/storage-classes/... . Almost everything in a k8s cluster can be ephemeral... except your data! But they make the defaults insane and it eats a lot of people's lunches. It's honestly very sad. I fortunately haven't "lost" data in 25 years, but I still remember the pain when I made this kind of mistake.
persistentVolumeReclaimPolicy: Retain
Alternatively set the reclaimPolicy on the StorageClass https://kubernetes.io/docs/concepts/storage/storage-classes/... for retaining all PVs of that SC after deletion of a PVC: reclaimPolicy: Retain
If Lily Cohen would have used either, recovery would have been possible. Of course then Lily would have to explicitly delete the PVs to prevent getting billed for the storage.Re: Total data loss after botched GitOps and failed backups
#70> We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data This is why for non hobby stuff I advocate for RDS, or its counterpart on your favorite provider. Running a production db on Kubernetes is looking for trouble unless you really,…