Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Total data loss after botched GitOps and failed backups

firefish.social

Post reply on HN
Print view View original

111–120 of 176 posts

Re: Total data loss after botched GitOps and failed backups

#111

Post by tamimio »

TekMol wrote: post #83

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world. None of the following is something which ever crossed my path: - GitOps repository cleanup - yaml manifests that create our namespaces - ArgoCD - Helm deployments - Persistent Volume Claims - Velero - Restic - PVC block volume d…

It’s just mostly Kubernetes stuff, totally not recommended, just stick with VM or use proxmox in your deployments and containers where you need to.

Re: Total data loss after botched GitOps and failed backups

#112

Post by testemailfordg2 »

One learning for all of us that I see every other day at work causing issues, the testing is not planned or not done based on what is present in production but something which makes the tester feel...Works for me.... Probably need a similar solution like Docker for this works for me disease in testing.

Re: Total data loss after botched GitOps and failed backups

#113

Post by tamimio »

Honestly, it shouldn’t feel that bad, what was the loss again here? A bunch of users and their posts/memes?! They can create other users it should not be an issue. The real fuckups IMO are stuff that involves people’s lives, things like industrial automation, robotics, autonomous vehicles, aircraft systems and what not, unfortunately, usually you don’t see such an accountability like in OP when things go south in those domains.

Re: Total data loss after botched GitOps and failed backups

#114

Post by sofixa »

nvm0n2 wrote: post #107
sofixa wrote: post #89

Earlier quoted context omitted.

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite co…

> GitOps is a philosophy which makes sense even on small projects. Er, does it? Root cause of this catastrophic dataloss incident is that in "GitOps" none of the traditional safety checks can be implemented. In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended. There…

> In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended

Depends on how you go about it, a wrong pathed mv could erase your data without a warning.

> Because git is intended for source code and not as a way to make stateful changes to servers, there are no features for that. If you push a commit that didn't do what you mean, it will just blindly do it.

There are no features in git itself for that, but in the way I usually implement GitOps, with a CI/CD system, you can easily have a manual "check this terraform plan's output to make sure you aren't doing anything crazy, and manually click this button to approve" step.

Re: Total data loss after botched GitOps and failed backups

#115

Post by Hakkin »

gnyman wrote: post #87
SeanAnderson wrote: post #49

Off-topic, but this website has some pretty crazy dark-patterns! I went to the user's profile and right-clicked their photo. The contextmenu action is intercepted and replaced with a custom menu mimicking the browser's context menu. This fake context menu has the option "Open in Window". I click it. The website doubles-down on the charade and opens a "popup" inside itself - maximize/minimize icons in the action bar a…

That's not a "dark pattern", it's just the result of of making the browser a OS and giving websites free rein of anything inside. There is an aptly named browser extension to help combat this called StopTheMadness [1] , I recommend it. Let's try to save the term Dark Patterns for the things that that deserve it, where the goal is to steer the user to make a unfavourable choice at the benefit of someone else. https://…

Firefox has a built-in method to get around context menu blocking by holding down the "shift" key while right clicking. I wish there were similar built-in overrides for things like blocking selecting text and copying, you commonly find stuff like this on corporate blogs or lyric sites where they want to "protect" their text content, of course you can just get around it by opening the page HTML and copying from there.

Re: Total data loss after botched GitOps and failed backups

#116

Post by RetroTechie »

andruby wrote: post #62
scarface_74 wrote: post #14

The only data that I really care about are my photos and videos and they are backed up to four different providers. But this also made me realize how upset I would be if my blog over at micro.blog was accidentally deleted. It’s more of a journal of my digital nomadding with my wife across the US than anything else. I immediately went over and started a JSON export in bar (?) format.

If it is a public blog, make sure the url is in the wayback machine as an extra external backup

Archive.org often captures sites excluding certain media like images or video.

So you'd think of that as an extra backup. But in reality might be just html, text-based stuff, or some other subset of file types.

Re: Total data loss after botched GitOps and failed backups

#117

Post by CommanderData »

hnarn wrote: post #108

I’m trying very hard to not be flippant here, but I can’t shake the feeling that this Kubernetes norm has to end. I’m not saying Kubernetes needs to disappear, but people need to stop treating it as the new normal, as if VM:s and config management is somehow an outdated and incapable alternative. To me, this is an example of the complexity of Kubernetes coming back to blow your foot off. Remember, Kubernetes exists t…

If the application is containerised, like most are these days, how do you propose running these on said VM?

If we're talking about something a little more complicated then a single container, then you arrive at the same problem Kubernetes is solving.

Re: Total data loss after botched GitOps and failed backups

#119

Post by nathancahill »

Hakkin wrote: post #115
gnyman wrote: post #87

Earlier quoted context omitted.

That's not a "dark pattern", it's just the result of of making the browser a OS and giving websites free rein of anything inside. There is an aptly named browser extension to help combat this called StopTheMadness [1] , I recommend it. Let's try to save the term Dark Patterns for the things that that deserve it, where the goal is to steer the user to make a unfavourable choice at the benefit of someone else. https://…

Firefox has a built-in method to get around context menu blocking by holding down the "shift" key while right clicking. I wish there were similar built-in overrides for things like blocking selecting text and copying, you commonly find stuff like this on corporate blogs or lyric sites where they want to "protect" their text content, of course you can just get around it by opening the page HTML and copying from there.

TIL. Thanks for this tip.

Re: Total data loss after botched GitOps and failed backups

#120

Post by hyperman1 »

Backup and restores are hard.

Last week, I upgraded my home laptop's SSD. I decided to take out the old one, do a fresh install on the new one, and restore the backup from a Synology NAS.

The good news: Backup seems complete. It's not the first time I migrated like this, and an older hardware upgrade went a lot less well, so I am happy.

The bad news: It was far from a trouble free process. I succeeded in the end, but had to restore a few times because the first tries were botched and there were some gotchas to learn.

Post reply on HN