Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Total data loss after botched GitOps and failed backups

firefish.social

Post reply on HN
Print view View original

141–150 of 176 posts

Re: Total data loss after botched GitOps and failed backups

#141

Post by SeanAnderson »

gnyman wrote: post #87
SeanAnderson wrote: post #49

Off-topic, but this website has some pretty crazy dark-patterns! I went to the user's profile and right-clicked their photo. The contextmenu action is intercepted and replaced with a custom menu mimicking the browser's context menu. This fake context menu has the option "Open in Window". I click it. The website doubles-down on the charade and opens a "popup" inside itself - maximize/minimize icons in the action bar a…

That's not a "dark pattern", it's just the result of of making the browser a OS and giving websites free rein of anything inside. There is an aptly named browser extension to help combat this called StopTheMadness [1] , I recommend it. Let's try to save the term Dark Patterns for the things that that deserve it, where the goal is to steer the user to make a unfavourable choice at the benefit of someone else. https://…

I still feel this is a dark pattern. It's attempting to keep engagement on the website and to prevent direct linking of underlying images. Direct linking images costs the server without providing sufficient marketing. You see a lot of different techniques employed to prevent image interaction. This is one such option.

The fact that they then implemented "Open in Window" rather than allowing the browser to handle it means they know they're affecting functionality the user wants and are choosing to provide an option that isn't the browser default. Why? It's certainly to their benefit. If it were to the user's benefit then it would be standard across all websites rather than provided by the browser by default.

Re: Total data loss after botched GitOps and failed backups

#143

Post by hdjjhhvvhga »

CommanderData wrote: post #117
hnarn wrote: post #108

I’m trying very hard to not be flippant here, but I can’t shake the feeling that this Kubernetes norm has to end. I’m not saying Kubernetes needs to disappear, but people need to stop treating it as the new normal, as if VM:s and config management is somehow an outdated and incapable alternative. To me, this is an example of the complexity of Kubernetes coming back to blow your foot off. Remember, Kubernetes exists t…

If the application is containerised, like most are these days, how do you propose running these on said VM? If we're talking about something a little more complicated then a single container, then you arrive at the same problem Kubernetes is solving.

> If the application is containerised, like most are these days, how do you propose running these on said VM?

As far as I can tell, this kind of load can be perfectly served by a €40 bare metal Hetzner instance (together with a Minecraft server and what not) so even a simple Docker Compose would do.

Re: Total data loss after botched GitOps and failed backups

#144

Post by fhaldridge7 »

diarrhea wrote: post #133
Gud wrote: post #128

Earlier quoted context omitted.

Reduce your abstractions and reduce your dependencies for a more stable environment. It will give you greater control while you at the same time don’t need to relearn everything every 5 years. Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine? This fatal scenario could have been easily avoided with a basic rsync cronjob to rsync.net or another $10 vi…

> Why does it have to be “containerised” at all? Why can’t the software be directly installed on a *nix virtual machine? I am very interested in this upcoming space. I had set up devbox.io for local development, and it even offers the concept of services. Like a docker compose for local development, just directly on the host. Nasty and hairy stuff like Python is still best shoved into a container to be done with it,…

I tried cloud-hypervisor recently and it booted the VM in 70ms, but I have no idea how to deploy this. We already have a deployment workflow for OCI images, doing this for VMs would mean a big refactoring

Re: Total data loss after botched GitOps and failed backups

#145

Post by manifoldgeo »

tamimio wrote: post #113

Honestly, it shouldn’t feel that bad, what was the loss again here? A bunch of users and their posts/memes?! They can create other users it should not be an issue. The real fuckups IMO are stuff that involves people’s lives, things like industrial automation, robotics, autonomous vehicles, aircraft systems and what not, unfortunately, usually you don’t see such an accountability like in OP when things go south in tho…

It sounds like this failure led to the complete destruction of 3 online communities. While this might not be as bad as, say, a fatal multi-car accident on the highway, I bet it led to some heavy grief as users mourned the losses of their communities.

It might not sound like a big deal because it's just "a bunch of users and their posts/memes", but many people get the vast majority of their social interactions through social media. A sudden, total loss of such a site could be devastating to someone's social life. They might've lost contact with an intimate friend group they knew only via aliases with no other way to reach out.

Even for older generations like Baby Boomers, life is an increasingly online experience:

  - How many senior citizens are reliant on Facebook for tons of their daily interactions with old friends? Probably a lot!

  - Entire degrees from reputable universities can be earned fully remotely, and the transcripts, curricula, forum posts, and even proof that you earned such a degree exist in databases and distributed systems vulnerable to the exact same types of flaws outlined in the OP.

  - Dating apps like Tinder / Bumble connect people who form long-lasting and loving relationships, and the sudden loss of such platforms could quickly end your potential lifelong romance.

  - LinkedIn networks are of huge importance to some for continued, gainful employment. Are there memes there? Sure! But does that mean it's "just a bunch of posts and memes" whose loss would be fine? Hardly, in my opinion.
My use of the above platforms as examples is not an endorsement of them, BTW. I just wanted to illustrate that the position you're defending will become more and more indefensible as people's lives become increasingly enmeshed in the digital world. Socializing online is still socializing, and a loss can still be painful even if there are memes and usernames involved.

Re: Total data loss after botched GitOps and failed backups

#146

Post by Terretta »

nvm0n2 wrote: post #107
sofixa wrote: post #89

Earlier quoted context omitted.

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite co…

> GitOps is a philosophy which makes sense even on small projects. Er, does it? Root cause of this catastrophic dataloss incident is that in "GitOps" none of the traditional safety checks can be implemented. In normal sysadmin workflows, attempting to delete all your data will yield an "Are you sure?!" type message and you'll probably have to take explicit steps to confirm that this is really what you intended. There…

> Because git is intended for source code and not as a way to make stateful changes to servers, there are no features for that. If you push a commit that didn't do what you mean, it will just blindly do it.

git doesn't do anything (except keep versioned source of declarative state).

Instead, have a look at your state engine, and make it as safe as you care to.

> because infrastructure isn't a stateless function of your git repository

So we agree, there's your problem. Not git, but the state engine function.

While you don't need K8s, for GitOps to work you do need a structured approach for operating your infrastructure. These concepts can help:

https://operatorframework.io/about/

“It is relatively easy to manage and scale web apps, mobile backends, and API services right out of the box. Why? Because these applications are generally stateless, so scripts can scale and recover infrastructure from failures without additional knowledge.”

“A larger challenge is managing stateful applications, like databases, caches, and monitoring systems. These systems require application domain knowledge to correctly scale, upgrade, and reconfigure while protecting against data loss or unavailability. We want this application-specific operational knowledge encoded into software … to run and manage the application correctly.”

Start by looking at the Operator Capability Level diagram here:

https://sdk.operatorframework.io/docs/overview/

You can iterate your ability to operate from desired state through those stages, with focus on the ones that hurt the most (by risk or by repetition).

Re: Total data loss after botched GitOps and failed backups

#147

Post by fhaldridge7 »

jakelazaroff wrote: post #33
sp332 wrote: post #18

Earlier quoted context omitted.

The point of the tooling is that you describe what you want your deploy to look like, and it updates the deploy to match the description. If you delete something from the description and it stays running, that would be very confusing.

I haven’t used their particular tool (ArgoCD) but the ones I have used include an option to keep specific pieces of infrastructure around even when it’s deleted from the description. That’s absolutely what you should be doing for anything that stores data.

ArgoCD will also not delete resources by default when the file is removed from the git repo. The author explicitly enabled auto-pruning.

Re: Total data loss after botched GitOps and failed backups

#148

Post by JohnMakin »

hnarn wrote: post #108

I’m trying very hard to not be flippant here, but I can’t shake the feeling that this Kubernetes norm has to end. I’m not saying Kubernetes needs to disappear, but people need to stop treating it as the new normal, as if VM:s and config management is somehow an outdated and incapable alternative. To me, this is an example of the complexity of Kubernetes coming back to blow your foot off. Remember, Kubernetes exists t…

It's hard to blame kubernetes as the issue here. The way EKS is implemented, it's basically EC2's running a bunch of containers with a convenient control plane and network interface abstracted over them.

If they had used EC2's/VM's, they could have still run into the same issue by blowing away a host's local volume they were never actually backing up. I have administrated social media websites (not this software, mainly discourse) and have seen this exact thing happen. Luckily, the backups worked.

Re: Total data loss after botched GitOps and failed backups

#149

Post by omniglottal »

bentinata wrote: post #67
SeanAnderson wrote: post #49

Off-topic, but this website has some pretty crazy dark-patterns! I went to the user's profile and right-clicked their photo. The contextmenu action is intercepted and replaced with a custom menu mimicking the browser's context menu. This fake context menu has the option "Open in Window". I click it. The website doubles-down on the charade and opens a "popup" inside itself - maximize/minimize icons in the action bar a…

It's not a website anymore. It's a web app. Might be related: I recently joined a company using Flutter web and notices similar stuff. Yes, the software engineers are wondering: "Why do we reimplement browser stuff?" But the business ask for it and we don't hear complaints from the end user.

"We don't hear complaints" != "there are no complaints"

Re: Total data loss after botched GitOps and failed backups

#150

Post by hdjjhhvvhga »

hnarn wrote: post #123
CommanderData wrote: post #117

Earlier quoted context omitted.

If the application is containerised, like most are these days, how do you propose running these on said VM? If we're talking about something a little more complicated then a single container, then you arrive at the same problem Kubernetes is solving.

It's a very complex discussion, almost complex enough to be unsuitable for text, because for every reply ten more "what-ifs" seem to open up. It's hard to be general enough that what you're saying is useful, while being specific enough that you're actually answering the question. If we're talking about a multi-container application with moving parts, then yes, Kubernetes is suitable exactly because it was created to…

You have omitted the crux of the issue: the whole point of GitOps is that you can always roll your deployments back, to any particular commit if needed. The very fact that GitOps was used against itself means it was set up incorrectly.
Post reply on HN