Total data loss after botched GitOps and failed backups

I have setups where the container orchestration was done completely by Terraform or Puppet. Docker Compose also comes a long way if you are not running a FAANG webscale service. If you manage your VM's basics with configuration management and infrastructure as code anyways, it's easy to bolt containers onto that and treat them like any other installable piece of code. Kubernetes does bring a nice abstraction if you have DevOps teams and separate Ops for your bare metal or Cloud.

It's a very complex discussion, almost complex enough to be unsuitable for text, because for every reply ten more "what-ifs" seem to open up. It's hard to be general enough that what you're saying is useful, while being specific enough that you're actually answering the question.

If we're talking about a multi-container application with moving parts, then yes, Kubernetes is suitable exactly because it was created to resolve the issues arising from that architecture. But I don't think it's fair to start a general discussion about architecture from a point of "Step 0: The application is made to run in Kubernetes".

By the way, is the statement that "most applications today are containerized" really true? I would claim that you can only count applications into this category that are published as "containerized by default", and to me that certainly doesn't seem true. I've encountered a few, but they're exceedingly rare. If you also count "applications that are available as a container", then sure, but then the statement should be that "most applications today *can* be containerized", or they are "available" in containerized form.

Whether something is "more complicated than a single container" depends not only on the application, but how you choose to deploy it. If you take a hypothetical web application running nginx for PHP with a Postgres backend, you can deploy both of those as containers, but you certainly don't have to. You could skip using containers entirely, or you could deploy the nginx component as unmanaged containers as if they were an RPM/DEB-package, while not involving containers for your database at all. Yes, it requires more configuration for all the things that are no longer "magic", but with static infrastructure the work you put in comes back to you as "simplicity": the moving parts are easier to understand, which means problems are easier to troubleshoot. Yes, you will lose "auto-healing", but you also lose "auto-nuking", and so on.

I want to be clear again that I'm not saying Kubernetes is objectively bad, but I am saying that I think a lot of people using it have not considered whether it's a suitable solution for their current problems (never-mind imaginary, future problems).

To me, it's like comparing a lawn mower to the space shuttle, and then trying to motivate building the latter by saying that a lawn mower will never be able to get into space: if you're running a web site that is looking at a few hundred thousand users, you're still (probably) not "going to space".

Fundamentally, I think the over-use of Kubernetes is nothing more than an example of premature optimization. The idea goes something like: "If we become the most successful /thing/ in the world, the infrastructure is already made in such a way that we can scale" -- but there are no decisions when it comes to infrastructure where you get the pros but can escape the cons, and those cons for Kubernetes specifically seem to rarely be discussed out of fear of sounding like a Luddite.

Most of production EBS volumes will be with deletion protection on. Also AWS asks you to explicitly type “delete” when you delete things.

I’d blame cloud provider for making it too easy to destroy data and not having built-in backups. But then this is what cheap cloud providers do.

There is!

You move on to another website.

Voting with your feet is the strongest veto we have as web users.

I think this is happening because we as a collective are using advertisement based websites with traffic funneled to them by advertisement network operators.

Don’t let Googles algorithms decide what website you visit. Find alternate search engines and indexes.

Well I guess I'll share my opinion too. K8s is not just for $BIGNAME companies. Personally I find k8s to be excellent for small shops because it has trivialized the install of large complex apps, opening doors for small groups to have a bigger impact. K8s is simply a package manager for complex deployments.

The learning curve for k8s may be a bit steep initially, but it covers all the areas you _should_ have some understanding of before tackling a large complex install of a web application and puts them in one place: manifests. The best Helm configs provide all the toggles you would need for complex deployments, but provide reasonable defaults (e.g. Mastodon [1]). Literally the application developers are also building the deployment _for_ you. It can't be overstated enough, because of what use is an app if you can't deploy it?

K8s and tools like ArgoCD also encourage some of the best modern practices with IaC. I don't need to hunt for every config file someone might have tinkered with to solve a problem on a server (and hopefully made a doc comment I can find to reproduce it). Every change is there in the manifests. There are OS variations, config variations, etc. and I just don't care about any of that! Just that k8s can run on it and use the resources. The infrastructure and git history is self documenting, allowing me to come into any small team and understand where they are at.

You said orchestration can take a lot of resources: meh. I trust something like k8s to use all of my compute resources more efficiently than a half dozen servers that are probably not sized correctly anyways for the individual components. Digital Ocean will run the control plane for free for example. What the orchestration overhead gives you is trivial reproduction anywhere. It really has nothing to do with "crazy scalability". It's the ability to tear down and bring up the whole cluster in one command. For example you can easily reproduce your full system locally for testing. Good luck doing that reliably with a bunch of servers without making it your full time job and making mistakes constantly.

Re pgsql backups: it's really not any more difficult on k8s. Maybe the problem is too many options: your storage class could do it for you with snapshots, you could have a script connect to the pod and run pg_dump, you could use a third party app like pgadmin4, or you could have sidecars do the work for you (assuming it is configured correctly too). In all cases you need to put the backup data somewhere. You could also use a managed postgres service and completely separate it from your cluster.

Back to the blog post: really the only problem the admin person had was not understanding the default settings of their storage class. Unfortunately that was also the most devastating. They could have also configured ArgoCD to not delete unknown manifests (opening up options to violate IaC). There is an information gap there, there is a gap in testing and backup strategy, and I hope others can learn and avoid getting bitten by it. But this blog post is not the reason to not use k8s. With some, honestly trivial, tweaks, they could still be supporting their community with these apps.

They said they used Vultr's k8s service, and they provide storage solutions also [2]. Did they contact Vultr to see if they implement some recovery safety for them? 30 day deletion policy, etc.? They probably knew about the problem in minutes, and data might have been or is recoverable in the backend.

[1] https://github.com/mastodon/chart/blob/main/values.yaml

[2] https://www.vultr.com/docs/vultr-kubernetes-engine/