Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Total data loss after botched GitOps and failed backups

firefish.social

Post reply on HN
Print view View original

91–100 of 176 posts

Re: Total data loss after botched GitOps and failed backups

#91

Post by politelemon »

SeanAnderson wrote: post #49

Off-topic, but this website has some pretty crazy dark-patterns! I went to the user's profile and right-clicked their photo. The contextmenu action is intercepted and replaced with a custom menu mimicking the browser's context menu. This fake context menu has the option "Open in Window". I click it. The website doubles-down on the charade and opens a "popup" inside itself - maximize/minimize icons in the action bar a…

Even trying to long press and copy some of the domain names that the author mentioned, was impossible. They've (who is they here?) implemented a context menu that interpreted the wrong link and offers to open it in a new tab.

Re: Total data loss after botched GitOps and failed backups

#92

Post by sofixa »

marcus_holmes wrote: post #79
mkl95 wrote: post #66

> We use #Velero to capture backups of our cluster every 6 hours. From what I had seen our backups had been running successfully. I discovered once the incident started that backups had captured everything but the Persistent Volume Claim data This is why for non hobby stuff I advocate for RDS, or its counterpart on your favorite provider. Running a production db on Kubernetes is looking for trouble unless you really,…

We had a database downtime on RDS. I queried it, because we were paying good money for a failover database. We were told that the downtime was because the database was being updated, and all databases are upgraded at the same time so the failover was also being upgraded. Never using RDS again. I'll stick with running Postgres on something I can actually manage myself.

That sounds unlike my experience with AWS RDS, and I don't think there would be many users of it if it were generally true. When was this and was it AWS?

Re: Total data loss after botched GitOps and failed backups

#93

Post by CoolCold »

TekMol wrote: post #83

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world. None of the following is something which ever crossed my path: - GitOps repository cleanup - yaml manifests that create our namespaces - ArgoCD - Helm deployments - Persistent Volume Claims - Velero - Restic - PVC block volume d…

You speak is like from one's of that sysadmins breed may speak. It's a heresy nowdays not to use kubernetes/not write yaml.

The only even more heresy thing could be - run web applications on baremetal servers!

Re: Total data loss after botched GitOps and failed backups

#94

Post by sofixa »

Kadin wrote: post #64
havnagiggle wrote: post #46

Earlier quoted context omitted.

IMO it's because of this default in k8s storage class: https://kubernetes.io/docs/concepts/storage/storage-classes/... . Almost everything in a k8s cluster can be ephemeral... except your data! But they make the defaults insane and it eats a lot of people's lunches. It's honestly very sad. I fortunately haven't "lost" data in 25 years, but I still remember the pain when I made this kind of mistake.

That explains why the K8S defaults are bad, but why use K8S at all here? Personally it seems like a lot of shade-tree server admins are way too eager to bust out much more tooling than is actually necessary for many tasks, just because it's the way that $BIGNAME company does it. Realistically, most people running a personal or small community's IT system aren't going to need the sort of crazy scalability that contain…

You kind of explained it yourself. You need to run a number of components, keep track of all of them, be able to update them, scale them, make sure they're healthy and restarted when needed, etc. This is of course possible in a number of ways, but not trivial. You're basically describing an orchestrator such as Kubernetes or Nomad. Especially with an already existing Helm chart covering all the deployment logic (what needs to be deployed, how many instances, health checks, etc.): https://github.com/mastodon/chart it's quite an easy choice instead.

Re: Total data loss after botched GitOps and failed backups

#95

Post by marcus_holmes »

sofixa wrote: post #92
marcus_holmes wrote: post #79

Earlier quoted context omitted.

We had a database downtime on RDS. I queried it, because we were paying good money for a failover database. We were told that the downtime was because the database was being updated, and all databases are upgraded at the same time so the failover was also being upgraded. Never using RDS again. I'll stick with running Postgres on something I can actually manage myself.

That sounds unlike my experience with AWS RDS, and I don't think there would be many users of it if it were generally true. When was this and was it AWS?

Copied the email we got from Amazon verbatim:

____

Hello Marcus,

Hope you are keeping well and safe. Thank you for reaching back out to me.

I can completely understand that this is not the behavior you expected with regards to engine version upgrades with multi-AZ configuration.

By design, when performing any version upgrade on a multi-AZ instance, the engine version is upgraded on the primary and the secondary at the same time resulting in both the primary and standby being unavailable. Unfortunately, it is not possible for the instance to failover in this scenario since the database level changes needs to be applied at the same time.

As mentioned in my previously correspondence, you do have the ability to opt-out of auto minor version upgrades if this does not suit your requirements [1]. By doing this, minor version updates will no longer be applied automatically and will have to be manually maintained.

As additional information, note that multi-AZ deployment does reduce the downtime in certain activities, however database engine version upgrade does not come under this. Multi-AZ deployments helps RDS automatically perform a failover in the event of any of the following:

1. An Availability Zone outage 2. The primary DB instance fails 3. The DB instance's server type is changed 4. The operating system of the DB instance is undergoing software patching 5. A manual failover of the DB instance was initiated using Reboot with failover

For more information on High Availability (Multi-AZ) deployments, kindly refer the below document for your reference [2].

I sincerely hope I was able to address your concerns Marcus. Please don't hesitate to reach back out to me and I will be happy to assist further as best I can.

Thank you for your time. Wishing you a great weekend ahead, and stay safe.

References:

[1] Automatically upgrading the minor engine version https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/USER_...

[2] Failover Process for Amazon RDS http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concep...

We value your feedback. Please share your experience by rating this correspondence using the AWS Support Center link at the end of this correspondence. Each correspondence can also be rated by selecting the stars in top right corner of each correspondence within the AWS Support Center.

Re: Total data loss after botched GitOps and failed backups

#96

Post by politelemon »

sofixa wrote: post #89
TekMol wrote: post #83

I have been building and running web applications which are used by millions of users for over 20 years now. But reading this post, I feel like I am looking into a completely different world. None of the following is something which ever crossed my path: - GitOps repository cleanup - yaml manifests that create our namespaces - ArgoCD - Helm deployments - Persistent Volume Claims - Velero - Restic - PVC block volume d…

I can see the point you're trying to make, but you're choosing an interesting way of conveying it, and I disagree. Restic is a backup tool. Velero is a backup tool for Kubernetes. Vultr is a low cost but still decent "cloud" provider. GitOps is a philosophy which makes sense even on small projects. None of those are "wrong" or "overcomplicated" options. The elephant in the room is Kubernetes, which is indeed quite co…

> You just have to be aware of the risks the complexity brings.

I'd say most are not aware. It is not enough to say that people should "just" be aware. That is something said from a position of knowledge and awareness, which helps nobody.

Completely agree with the parent comment, simplicity is the first thing that should be reached for. K8s ought to be dismissed by default, because then you have to justify its inclusion. That's probably a way to increase awareness before plunging in.

Re: Total data loss after botched GitOps and failed backups

#97

Post by sofixa »

livueta wrote: post #28
noobermin wrote: post #22

Earlier quoted context omitted.

This is a dumb question, but why can't people just make things simpler? Tools are supposed to make work easier and make things more efficient, but this sort of complexity just seems to hurt, doesn't it?

I wonder if it has to do with mismatches between -as-cattle and -as-pets philosophies and usecases. From the post, it looks like they were using k8s. It's hard to tell given the, you know, total data loss, but it looks like the instance had at max triple-digit users. In my experience running small community stuff like this, the -as-cattle tooling, even though I'm familiar with it from work, is way overkill and introd…

> Some of that may be on devs, actually: distribution-as-docker-container at the expense of real packaging seems to be on the rise.

Because it eliminates a whole host of "works on my computer" "which version of X dependency library do you have installed" problems nobody wants to deal with.

Re: Total data loss after botched GitOps and failed backups

#98

Post by bomewish »

gabereiser wrote: post #52
appplication wrote: post #40

Earlier quoted context omitted.

I want to feel bad for them, and I do, but really I’m not sure what you expect. They never tested their backup. They tested their ability to manually back up, but I’m not sure why they would ever think to conflate that with automatic processes. If they had automatic backups, why were they relying on manual ones for testing?

Yeah. Kudos to being open about it but this is an amateur mistake. I ran cloud infrastructure for large companies, small companies, and small companies with large data. Disaster Recovery, Disaster Response, Data Backups, Time to restore, etc are all table stakes to running anything in “production”. They tested their ability to do a backup, but never went through with actually backing it up - automating it - let alone…

but why didn't they just use some normal setup like a postgres database or something? Why all the intense complexity? That makes it less OK

Re: Total data loss after botched GitOps and failed backups

#99

Post by dusted »

While my knee-jerk reaction is to blame bad backup practices, that's not what I believe happened here. Not entirely at least.

The setup is too complex to easily verify and restore.

Yes, a bespoke box is annoying and "inelegant" but it also works, and if it's backed up following even 80's tier best practice, it can be restored by anyone with a pulse.

Re: Total data loss after botched GitOps and failed backups

#100

Post by mannyv »

pdntspa wrote: post #4

Wow... tooling that deletes a bunch of stuff because a manifest file is missing. Just... wow. I feel for the admin here. This seems equivalent to an accidental `sudo rm -rf /`.

Terraform will do this. It's just like a command line, but for your environment.
Post reply on HN